Introduction

In some of our earlier articles we’ve explained the setting up of chrooted sftp and chrooted ssh accounts along with the setup of an ftp server using vsftpd. In today’s automation driven enterprise infrastructure environments we may often find ourselves in situations where we need to automate the transfer of files using the ftp/sftp protocol. Setting up passwordless tranfer of files using sftp is farily stragithforward.  Since sftp is based on the ssh protocol we can generate and use ssh keys to setup passwordless authentication to allow for automated file tranfers. In this article we will demonstrate three techniques using which you can automate the transfer of files using FTP protocol as well.

Method 1: Using wget
The wget command provides options to connect to the ftp server and download a file while specifying the credentials on the command line. In the below example, we connect to the ftp server with IP address 172.31.18.17 using the user name sahil and password as L#giN@123 and download the file download.txt.

[root@sahilsuri0081 ~]# wget --user=sahil --password='L#giN@123' ftp://172.31.18.17/download.txt
--2018-07-27 17:39:40-- ftp://172.31.18.17/download.txt
=> ‘download.txt’
Connecting to 172.31.18.17:21... connected.
Logging in as sahil ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD not needed.
==> SIZE download.txt ... done.
==> PASV ... done. ==> RETR download.txt ... done.

[ <=> ] 0 --.-K/s in 0s

2018-07-27 17:39:40 (0.00 B/s) - ‘download.txt’ saved [0]

[root@sahilsuri0081 ~]# ls -l download.txt
-rw-r--r--. 1 root root 0 Jul 27 17:39 download.txt
[root@sahilsuri0081 ~]#

The shortcoming of using this method is that we have to specify the credentials in the command itself so it’s not very secure. The wget command only allows us to download files from the ftp server. It does not allow us to upload files.
Although there is a utility named wput which works like wget but instead of downloading files it’s purpose is to upload files.

Method 2: Using curl
The curl command provides us options to download as well as upload files in an automated manner. While using the curl command we specify the username/password by using the -u option followed by the username and password delimited by a colon. We use the -O option specify the FTP server URL. Given below is an example.

[root@linuxnix ~]# curl -u sahil:L#giN@123 -O ftp://172.31.18.17/download.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20 100 20 0 0 430 0 --:--:-- --:--:-- --:--:-- 444
[root@linuxnix ~]# ls -l download.txt
-rw-r--r--. 1 root root 20 Jul 28 10:30 download.txt
[root@linuxnix ~]#

Note: Curl will not download the file if the file is empty.

To upload a file using the curl command we use the -T option to specify the file name to be uploaded and we do not use the -O option to indicate the URL in this case. Given below is an example:

[root@linuxnix ~]# curl -u sahil:123 ftp://172.31.18.17/ -T file.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 69 0 0 100 69 0 1503 --:--:-- --:--:-- --:--:-- 1533

Method 3: Using here documents
In the third and final technique we’ll rely on here documents. A here document is a special-purpose code block. It uses a form of I/O redirection to feed a command list to an interactive program or a command, such as ftp, cat, or the ex text editor. A limit string delineates (frames) the command list. The special symbol << precedes the limit string. This has the effect of redirecting the output of a command block into the stdin of the program or command.  It is similar to interactive-program < command-file, where command-file contains To demonstrate the automation of the FTP file transfer process using here documents I’ve written a small script shown below

[root@linuxnix ~]# cat ftp.bash
ftp -in 172.31.18.17 << EOF > ~/ftp-session-log.txt 2>&1
user sahil L#giN@123
get download.txt
bye
EOF
[root@linuxnix ~]#

In the above script we’ve used the -i and -n options with the FTP command wherein -i prevents interactive login prompts and -n option prevents auto-login attempts. Therefore we can specify all the FTP commands that we intend to execute during our FTP session within the body of the here document. In the above script we are logging in to our FTP server as the user sahil by providing the user name and password and then we download a file named download.txt. After this we disconnect the session. The sessions logs are being sent to a file named ftp-session-log.txt. Let’s execute this script and then we can review the contents of this file.

[root@linuxnix ~]# bash -x ftp.bash
+ ftp -in 172.31.18.17
[root@linuxnix ~]# cat ftp-session-log.txt
WARNING! 1 bare linefeeds received in ASCII mode
File may not have transferred correctly.

Conclusion

In this article we demonstrated three different techniques to automate our FTP file transfers. From among the methods listed we would recommend using here documents as we can implement the setup in shell scripts and it would provide more flexibility.

Post Views: 5,997
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.