Cloudbase database migration scheme

Function is introduced

Migration between cloud development database environments has always been a difficult problem , although SDK For a single set export and import, But to achieve the migration of the entire database is just 100 The first of the steps , The scheme is to introduce a method of A The environment database is migrated to B The idea of environmental database , For reference only .

Preparation needed

1. Deploy the functions under the folder to the corresponding environment
2. Created A Environment and B Environmental Science
3. If the migrated environment is under another account , You need another account secretID and secretKey, And in init The two fields are added to the environment
4. You can operate directly on the cloud development console , There is no need to initiate a call
5. Set the timeout of the function to 900 second .

Resources used

1. Cloud functions for both environments
2. Cloud databases for both environments
3. Cloud storage for the target environment

Function introduction

migrate

Migration functions , To be deployed to In a migrated environment

Remember to revise demo Environment in ID For your own environment ID

It is necessary to initiate the call , No parameters required .

It will be called exportDatabase Function to export data , And get an array of return values , The data format is roughly as follows

 [
    {
      "result": {
        "CollectionName": "test2Collection",
        "exportRes": {
          "JobId": 102223492,
          "RequestId": "332f1595-9b64-4a42-a6b3-a0ca1e0b50e5"
        }
      },
      "requestId": "ff84c3dc-d3fa-11eb-bb41-525400be5245"
    },
    {
      "result": {
        "CollectionName": "testCollection",
        "exportRes": {
          "JobId": 102223493,
          "RequestId": "92b8e0b5-e4a9-43bd-b9ef-37e6abca8b83"
        }
      },
      "requestId": "004ba931-d3fb-11eb-a0a6-52540096f86f"
    }
  ]

And then call copyDatabase, Migration

exportDatabase

The export function , To be deployed to In a migrated environment

By migrate call

copyDatabase

To be deployed to In a migrated environment

By migrate call

{
  JobIdList:Array
}

 This function will initiate a call importDatabase, If returns success, The migration is successful .

importDatabase

Import function , To be deployed to Migrate to the target environment

This function receives CollectionName Function for collection import , Users do not need to actively invoke , The function is copyDatabase call .

Sequence diagram

image.png

Complete steps

1. Deploy the function to the corresponding environment （ except importDatabase Is deployed to the target environment , The remaining three are migrated environments ）, Remember that cloud installation depends on .
2. call migrate（ It is recommended that the console directly call ）
3. Wait for the return value
4. Go to the new environment database to view the migration results

matters needing attention

1. because export This interface can only be called once per second , So to be on the safe side , At the code level, each set is exported every second . therefore migrate This function will take a long time to call , It is recommended to wait patiently （ Every set needs about 2 second ）, If the number of sets is large , It is recommended to adjust the function timeout upwards . If greater than 900 second , Consider using cloud hosting to implement .
2. migrate All logic will be executed at once , Long running time , Please be patient , You can do other things while waiting .
3. The export of each document is asynchronous and takes time , obtain migrate After the return value of , Please wait a while before calling copyDatabase, Since the size of each document is variable, the time is uncertain , But in copyDatabase Detection function is built in , If the call is initiated before the full export , This function will stop and prompt which item has not been exported yet , You can call .
4. return success after , Go to the console to check that the collection has been created , But the number of documents is 0 The situation of , This is because the import is an asynchronous event , Creating collections and importing documents are not done at the same time , Just wait for a while .
5. The number of documents is 0 Will be skipped .
6. If a system error occurs , Internal errors, etc , You can appropriately increase the cloud function running memory .
7. if migrate The function console appears ESOCKETTIMEDOUT Or other errors , But the other three functions all work well , That can be ignored , It depends on whether the database data of the target environment is migrated correctly .
8. The cloud storage of the target environment is borrowed as the medium , If it is too large, it can be deleted , The catalogue is /tmp/db-imports
9. Not rigorously tested , I don't know the performance limit yet , Current test 20+ aggregate , Hundreds of documents are OK .

Open source code

https://github.com/Readercyc/Cloudbase-Solution