Daniele Cammarata

Automating repetitive tasks can save a significant amount of time and ensure consistency. In this blog post, we’ll walk through the steps to set up a scheduled GitHub Action that executes a TypeScript script. This script will call an API, process the data, and upload the result to an S3 bucket.

Prerequisites

Before we start, ensure you have the following:

A GitHub account
An AWS account with access to an S3 bucket
Basic knowledge of TypeScript and GitHub Actions

Step 1: Create the TypeScript Script

First, let’s create a TypeScript script that fetches data from an API and uploads it to an S3 bucket. Save the following script as fetch_and_upload.ts in your repository:

import axios from 'axios';
import AWS from 'aws-sdk';
import { config } from 'dotenv';
import { v4 as uuidv4 } from 'uuid';

// Load environment variables from .env file
config();

// Replace with your API endpoint
const API_URL = "https://api.example.com/data";
const S3_BUCKET = "your-s3-bucket-name";

const fetchData = async (): Promise<any> => {
    const response = await axios.get(API_URL);
    return response.data;
};

const uploadToS3 = async (data: any): Promise<void> => {
    const s3 = new AWS.S3({
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
    });

    const fileName = `data_${new Date().toISOString()}.json`;
    const fileContent = JSON.stringify(data);

    const params = {
        Bucket: S3_BUCKET,
        Key: fileName,
        Body: fileContent,
        ContentType: 'application/json'
    };

    await s3.upload(params).promise();
    console.log(`Uploaded data to s3://${S3_BUCKET}/${fileName}`);
};

const main = async (): Promise<void> => {
    try {
        const data = await fetchData();
        await uploadToS3(data);
    } catch (error) {
        console.error("Error fetching or uploading data:", error);
    }
};

main();

This script does the following:

Fetches data from an API.
Uploads the fetched data to an S3 bucket with a timestamped filename.

Step 2: Set Up AWS Credentials

To allow your script to upload data to S3, you need to provide AWS credentials. Store these credentials as GitHub Secrets:

Go to your GitHub repository.
Navigate to Settings > Secrets and variables > Actions.
Click on New repository secret.
Add the following secrets:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY

Step 3: Create the GitHub Action Workflow

Now, let’s create a GitHub Actions workflow that runs the TypeScript script on a schedule. Create a new file in your repository at .github/workflows/scheduled_fetch.yml with the following content:

name: Scheduled API Data Fetch and Upload to S3

on:
  schedule:
    - cron: '0 * * * *'  # Runs every hour

jobs:
  fetch-and-upload:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Set up Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '16'

    - name: Install dependencies
      run: |
        npm install
        npm install typescript ts-node axios aws-sdk dotenv uuid

    - name: Run fetch_and_upload.ts
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: npx ts-node fetch_and_upload.ts

This workflow does the following:

Runs on a schedule (every hour in this example).
Checks out the repository.
Sets up Node.js.
Installs the required dependencies (typescript, ts-node, axios, aws-sdk, dotenv, and uuid).
Executes the fetch_and_upload.ts script with AWS credentials passed as environment variables.

We can change the scheduled time using crontab guru.

Step 4: Commit and Push Changes

Commit and push your changes to GitHub:

git add fetch_and_upload.ts .github/workflows/scheduled_fetch.yml
git commit -m "Add scheduled GitHub Action to fetch API data and upload to S3 using TypeScript"
git push origin main

Conclusion

By following these steps, you’ve automated the process of fetching data from an API and uploading it to an S3 bucket using GitHub Actions with TypeScript. This setup ensures that your data is consistently updated and stored without manual intervention. You can adjust the schedule and customize the script as needed to fit your specific requirements.