Why and how to setup an AWS Aurora Serverless database ?

Why and how to setup an AWS Aurora Serverless  database ?

In our AWS cloud native application development projects we often use native-cloud technology like AWS Lambda and AWS ECS Fargate to accelerate deployment and minimize costs. The power of both products is off course the (low) pay-per-use model but even better the low total operating expenditures (OPEX) in the lifecycle management of the application.

What we noticed in the initial cloud native applications we maintained from DevOps perspective is that quite often this meant two other components became relative expensive. First monitoring, logging, tracing and alerting in a distributed  application (that's for another blog) and the relational databases.

Why ? Because auto-scaling on the different RDS types is quite often not that advanced, leaving you with a scale-to-max setup on environments for the performance tests and production.

With the announcement of Aurora Serverless we started using this for a new project and later one started migrating existing applications. Why ? Because it has some unique features on the data layer

✔︎ Easy scaling (less OPS burden)
✔︎ Pay-per-use (quite unique as said)
✔︎ Accessible over HTTP (Data API)
✔︎ Authentication & authorization using IAM roles rather than database roles

An infra-as-code example

So below is an example CloudFormation template  how to configure the database. Because if you do it, you do it right. We first generate both the master and a system user for Lambda and store it in SecretsManager.

Resources:

  # SecretsManager
  # Generate the master user and store it
  SecretsManagerSecretMaster:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub 't10/${ENV}/rds/t10master'
      Description: !Sub 't10/${ENV}/rds/t10master'
      GenerateSecretString:
        SecretStringTemplate: '{"username": "t10master"}'
        GenerateStringKey: 'password'
        PasswordLength: 15
        ExcludeCharacters: '"@/\'
  SecretsManagerSecretTargetAttachmentMaster:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId: !Ref SecretsManagerSecretMaster
      TargetId: !Ref AuroraCluster
      TargetType: AWS::RDS::DBCluster          
  
  # LAMBDA user: manual create it within the Database using the master
  SecretsManagerSecretLambda:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub 't10/${ENV}/rds/t10lambda'
      Description: !Sub 't10/${ENV}/rds/t10lambda'
      GenerateSecretString:
        SecretStringTemplate: '{"username": "t10lambda"}'
        GenerateStringKey: 'password'
        PasswordLength: 15
        ExcludeCharacters: '"@/\<>'
  SecretsManagerSecretTargetAttachmentLambda:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId: !Ref SecretsManagerSecretLambda
      TargetId: !Ref AuroraCluster
      TargetType: AWS::RDS::DBCluster    
  SecretsManagerRotationScheduleLambda:
    Type: AWS::SecretsManager::RotationSchedule
    Properties:
      SecretId: !Ref SecretsManagerSecretLambda
      RotationLambdaARN: 'arn:aws:lambda:eu-west-1:xxxx:function:SecretsManagermysql-rotation' #generic
      RotationRules:
        AutomaticallyAfterDays: 30

The next segment will then create the security group, parameter group and the Aurora serverless database itself.

Resources:
  
  AuroraSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Sub 't10-${ENV}-sg-aurora'
      GroupName: !Sub 't10-${ENV}-sg-aurora'
      VpcId: vpc-123456
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 3306
          ToPort: 3306
          CidrIp: 10.0.0.0/8
      SecurityGroupEgress:
        - IpProtocol: '-1'
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub 't10-${ENV}-sg-aurora'    
  
  AuroraCluster:
    Type: AWS::RDS::DBCluster 
    Properties:
      BackupRetentionPeriod: 28
      DBClusterParameterGroupName: !Ref DBClusterParameterGroup
      Engine: aurora
      EngineVersion: 5.6.10a
      EngineMode: serverless
      ScalingConfiguration:
        AutoPause: false
        MinCapacity: 1
        MaxCapacity: 64
        # SecondsUntilAutoPause: 86400
      StorageEncrypted: true
      MasterUsername: !If
        - UseDbSnapshot
        - !Ref 'AWS::NoValue'
        - t10master
      MasterUserPassword: !If 
        - UseDbSnapshot
        - !Ref 'AWS::NoValue'
        - !Join ['', ['{{resolve:secretsmanager:', !Ref SecretManagerSecretMaster, ':SecretString:password}}' ]]
      DatabaseName: !If 
        - UseDbSnapshot
        - !Ref 'AWS::NoValue'
        - !Sub 't10${ENV}'
      DBClusterIdentifier: !Sub 't10-${ENV}-dbcluster'
      Port: 3306
      DBSubnetGroupName: !Sub 't10-${ENV}-dbsubnetgroup'
      SnapshotIdentifier: !If 
        - UseDbSnapshot
        - !Ref DBSnapshotIdentifier
        - !Ref 'AWS::NoValue'
      DeletionProtection: true
      VpcSecurityGroupIds:
        - !Ref auroraSecurityGroup
  
  DBClusterParameterGroup:
    Type: AWS::RDS::DBClusterParameterGroup
    Properties:
      Description: !Sub 't10-${ENV}-parametergroup'
      Family: aurora5.6
      Parameters:
        event_scheduler: 'ON'
      Tags:
        - Key: Name
          Value: !Sub 't10-${ENV}-parametergroup'
       

Going into details

We use a common AWS CloudFormation mechanism here to determine if we need to restore from an earlier snapshot by checking if the parameter DBSnapshotIdentifier is passed as input.

Conditions:
  UseDbSnapshot: !Not 
    - !Equals 
      - !Ref DBSnapshotIdentifier
      - ''

The power of Aurora Serverless is in this segment. By setting the ScalingConfiguration we let our database auto-scale between a minimum and maximum capacity. So no complex auto-scaling configuration with events, no provisioning of RDS instance types and clusters. This is it. In the below example the AutoPause feature is disabled which means the database is always there in the least minimum capacity (1 in the example). If you enable it the RDS will stop completely after 0 connection during the SecondsUntilAutoPause (set to the 24 hours maximum here). This is nice for PoC and personal accounts (to save money) but bare in mind that starting the database can take up to 60 seconds (personal experience).

   ScalingConfiguration:
        AutoPause: false
        MinCapacity: 1
        MaxCapacity: 64
        # SecondsUntilAutoPause: 86400

The scaling configuration result can then be something like the image below. Since it's pay-per-use operational cost can decrease drastically as we have seen for our cloud native applications.

Hope it helps!

References: