Access Google Cloud Storage (GCS)via SFTP

SFTP to a GCS Bucket

What is GCS?

Object Storage provided by GCP which can scale infinitely.

What is SFTP ?

SFTP, or Secure File Transfer Protocol, is a secure file transfer protocol that uses secure shell encryption to provide a high level of security for sending and receiving file transfers.

In order to achieve SFTP access to GCS Bucket we need the below components

1. Google Compute Engine (VM)
2. GCS Fuse (rpm package)
3. GCS Bucket
4. SFTP Client (WinSCP/Filezilla)

• Create a GCE Virtual Machine with the below configuration

1. OS = CentOs 7
2. Identity and API access →Access Scopes →Set access for each API and give Full Access to Storage

Setting access for writing to GCS Bucket

• Create a GCS Bucket
• SSH into GCE and execute the below Command[as root account] to create a user for SFTP

1. useradd -s /sbin/nologin sftpuser
2. passwd sftpuser
3. mkdir -p /var/sftp/upload
4. chown sftpuser:sftpuser /var/sftp/upload
5. chown root:root /var/sftp
6. chmod 755 /var/sftp
7. edit sshd_config as per the below instructions:

# find Subsystem sftp /usr/libexec/openssh/sftp-server and replace with Subsystem sftp internal-sftp

# add below lines to the end of the file

Match User sftpuser
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /var/sftp
AllowTcpForwarding no
X11Forwarding no

8. systemctl restart sshd

• Installing GCS Fuse [execute below commands]

sudo tee /etc/yum.repos.d/gcsfuse.repo > /dev/null <<EOF
[gcsfuse]
name=gcsfuse (packages.cloud.google.com)
baseurl=https://packages.cloud.google.com/yum/repos/gcsfuse-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

sudo yum install gcsfuse

• Mount the GCS Bucket

sudo gcsfuse -o allow_other -uid=<uid of sftp user> -gid=<gid of sftp user> -file-mode=777 -dir-mode=777 <Bucket_Name>

• Access the Bucket using SFTP

Now use the External IP of the GCE to connect to the GCS as an SFTP location

SFTP to the GCE instance

putting objects into the upload folder will directly copy the objects to GCS Bucket