There is quite a bit more to Amazon S3 than storing public files. In this article, we look into how we can manage private files. To do so, we learn how to set up a proper private Amazon S3 bucket and how to upload and access files. We use streams and generate presigned URLs with an expiration time. You can find the code from this series in this repository. Setting up Amazon S3 The first thing to do is to create a new bucket. This time, we intend to restrict access to the files we upload. Every time we want our users to be able to access a file, they will need to do it through our API. The IAM user that we’ve created in the previous part of this series has access to all our buckets. Therefore, all we need to do to start using it is to add the name of the bucket to our environment variables. .env # ... AWS_PRIVATE_BUCKET_NAME=nestjs-series-private-bucket /src/app.module.ts ConfigModule.forRoot({ validationSchema: Joi.object({ // ... AWS_PRIVATE_BUCKET_NAME: Joi.string().required(), }) }) Managing files through the API Once we have the above set up, we can start uploading files to our private bucket. When doing so, we want to save them in a similar way wheKolejny rok n dealing with public files. This time we won’t save the URL of the file, though. Let’s allow our users to manage some files. To do that, let’s create the entity of a private file. It needs to contain the id of the user. /src/privateFiles/privateFile.entity.ts import { Column, Entity, ManyToOne, PrimaryGeneratedColumn } from 'typeorm'; import User from '../users/user.entity'; @Entity() class PrivateFile { @PrimaryGeneratedColumn() public id: number; @Column() public key: string; @ManyToOne(() => User, (owner: User) => owner.files) public owner: User; } export default PrivateFile;Now we need to add information about the other side of the relationship. /src/users/user.entity.ts import { Entity, OneToMany } from 'typeorm'; import PrivateFile from '../privateFIles/privateFile.entity'; @Entity() class User { // ... @OneToMany( () => PrivateFile, (file: PrivateFile) => file.owner ) public files: PrivateFile[]; } export default User; If you want to know more about defining relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM We need to save the key so that we can access or delete our private files. Let’s create a separate service to manage them. /src/files/privateFiles.service.ts import { Injectable } from '@nestjs/common'; import { InjectRepository } from '@nestjs/typeorm'; import { Repository } from 'typeorm'; import { S3 } from 'aws-sdk'; import { ConfigService } from '@nestjs/config'; import { v4 as uuid } from 'uuid'; import PrivateFile from './privateFile.entity'; @Injectable() export class PrivateFilesService { constructor( @InjectRepository(PrivateFile) private privateFilesRepository: Repository, private readonly configService: ConfigService ) {} async uploadPrivateFile(dataBuffer: Buffer, ownerId: number, filename: string) { const s3 = new S3(); const uploadResult = await s3.upload({ Bucket: this.configService.get('AWS_PRIVATE_BUCKET_NAME'), Body: dataBuffer, Key: `${uuid()}-${filename}` }) .promise(); const newFile = this.privateFilesRepository.create({ key: uploadResult.Key, owner: { id: ownerId } }); await this.privateFilesRepository.save(newFile); return newFile; } } /src/files/privateFiles.module.ts import { Module } from '@nestjs/common'; import { TypeOrmModule } from '@nestjs/typeorm'; import { PrivateFilesService } from './privateFiles.service'; import { ConfigModule } from '@nestjs/config'; import PrivateFile from './privateFile.entity'; @Module({ imports: [ TypeOrmModule.forFeature([PrivateFile]), ConfigModule, ], providers: [PrivateFilesService], exports: [PrivateFilesService] }) export class PrivateFilesModule {}Once that’s done, we can use all of the above to upload private files for our users. /src/users/users.service.ts import { Injectable } from '@nestjs/common'; import { InjectRepository } from '@nestjs/typeorm'; import { Repository } from 'typeorm'; import User from './user.entity'; import { PrivateFilesService } from '../privateFIles/privateFiles.service'; @Injectable() export class UsersService { constructor( @InjectRepository(User) private usersRepository: Repository, private readonly privateFilesService: PrivateFilesService ) {} // ... async addPrivateFile(userId: number, imageBuffer: Buffer, filename: string) { return this.privateFilesService.uploadPrivateFile(imageBuffer, userId, filename); } } /src/users/users.controller.ts import { UsersService } from './users.service'; import { Controller, Post, Req, UploadedFile, UseGuards, UseInterceptors } from '@nestjs/common'; import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard'; import RequestWithUser from '../authentication/requestWithUser.interface'; import { FileInterceptor } from '@nestjs/platform-express'; import { Express } from 'express'; @Controller('users') export class UsersController { constructor( private readonly usersService: UsersService, ) {} // ... @Post('files') @UseGuards(JwtAuthenticationGuard) @UseInterceptors(FileInterceptor('file')) async addPrivateFile(@Req() request: RequestWithUser, @UploadedFile() file: Express.Multer.File) { return this.usersService.addPrivateFile(request.user.id, file.buffer, file.originalname); } }After doing all of the above, our users can start uploading private files. You can also implement deleting the files in a very similar way as in the previous part of this series. It might be worth checking if the currently logged in user is an owner of the file, though. Accessing private files Since the files we upload above are private, we can’t access them by simply entering a URL. Trying to do so will result in getting an error. There is more than one way to approach this issue. Let’s start with the most straightforward one. Fetching the file from Amazon S3 as a stream The first solution to the above issue is to send the file through our API. The most fitting way to do that is to pipe a readable stream that we can get from the AWS SDK to our response. Thanks to working directly with streams, we don’t have to download the file into the memory in our server. If you want to know more about piping streams in Node check out Node.js TypeScript #5. Writable streams, pipes, and the process streams The first thing to do is to get a readable stream of data from our Amazon S3 bucket. /src/privateFiles/privateFiles.service.ts import { Injectable } from '@nestjs/common'; import { InjectRepository } from '@nestjs/typeorm'; import { Repository } from 'typeorm'; import { S3 } from 'aws-sdk'; import { ConfigService } from '@nestjs/config'; import PrivateFile from './privateFile.entity'; import { NotFoundException } from '@nestjs/common'; @Injectable() export class PrivateFilesService { constructor( @InjectRepository(PrivateFile) private privateFilesRepository: Repository, private readonly configService: ConfigService ) {} // ... public async getPrivateFile(fileId: number) { const s3 = new S3(); const fileInfo = await this.privateFilesRepository.findOne({ id: fileId }, { relations: ['owner'] }); if (fileInfo) { const stream = await s3.getObject({ Bucket: this.configService.get('AWS_PRIVATE_BUCKET_NAME'), Key: fileInfo.key }) .createReadStream(); return { stream, info: fileInfo, } } throw new NotFoundException(); } }Now we need to make sure if the users should be able to download the file. /src/users/users.service.ts import { Injectable, UnauthorizedException } from '@nestjs/common'; import { InjectRepository } from '@nestjs/typeorm'; import { Repository } from 'typeorm'; import User from './user.entity'; import { FilesService } from '../files/files.service'; import { PrivateFilesService } from '../privateFIles/privateFiles.service'; @Injectable() export class UsersService { constructor( @InjectRepository(User) private usersRepository: Repository, private readonly filesService: FilesService, private readonly privateFilesService: PrivateFilesService ) {} // ... async getPrivateFile(userId: number, fileId: number) { const file = await this.privateFilesService.getPrivateFile(fileId); if (file.info.owner.id === userId) { return file; } throw new UnauthorizedException(); } }The interesting thing happens in the controller. Since we are working with streams directly, we need to access the Response object that NestJS uses under the hood. /src/users/users.controller.ts import { UsersService } from './users.service'; import { Controller, Get, Param, Req, Res, UseGuards, } from '@nestjs/common'; import JwtAuthenticationGuard from '../authentication/jwt-authentication.guard'; import RequestWithUser from '../authentication/requestWithUser.interface'; import { Response } from 'express'; import FindOneParams from '../utils/findOneParams'; @Controller('users') export class UsersController { constructor( private readonly usersService: UsersService, ) {} // ... @Get('files/:id') @UseGuards(JwtAuthenticationGuard) async getPrivateFile( @Req() request: RequestWithUser, @Param() { id }: FindOneParams, @Res() res: Response ) { const file = await this.usersService.getPrivateFile(request.user.id, Number(id)); file.stream.pipe(res) } } Above, we use the FindOneParams DTO for the purpose of validation. If you want to know more, check out API with NestJS #4. Error handling and data validation Generating presigned URLs Responding with