Stream Amplitude Export API to AWS S3
A lot of drama at work, this time I need to backup the Amplitude logs events to S3. The dumb idea, I think I can just export the all amplitude events to my local machine, and upload it to S3. And not, so dumb idea, I think, we can do it all at once: Streaming.
And thanks God, there is also the npm package already: s3-stream-upload. So this is all I have to do to send amplitude exports between startDate and andDate
1const AWS = require("aws-sdk");
2const Amplitude = require('amplitude');
3const UploadStream = require('s3-stream-upload');
4
5const AWSBUCKET = "<YOUR-AWS-BUCKET>";
6const accessKeyId = "<YOUR-AWS-ACCESS-KEY-ID>";
7const secretAccessKey = "<YOUR-AWS-SECRET-KEY-ID>";
8AWS.config.update({ accessKeyId, secretAccessKey });
9
10const s3 = new AWS.S3();
11
12let amplitude = new Amplitude(apiToken, { secretKey });
13
14let key = `amplitude_logs/${startDate}_${endDate}.zip`;
15
16amplitude.export({
17 start: startDate,
18 end: endDate
19 })
20 .pipe(UploadStream(s3, { Bucket: AWSBUCKET, Key: key}))
21 .on("error", function (err) {
22 console.error(err);
23 })
24 .on("finish", function () {
25 console.log("Uploaded: ", key);
26 });
You have to install all dependency first, of course.
npm install amplitude aws-sdk s3-stream-upload
And, by the way, in fact I should do more than above actually. I need to export 3 years Amplitude history, so need to create some async/thread call to make it feasible and faster at run.