Barrick Lab :: CommandLineBox

---+ Copying files to/from UT Box at the command line

If you use SSO (through UT Box), you need to first [[https://support.box.com/hc/en-us/articles/360043694574-Box-SSO-Working-with-External-Passwords][set up an external password]].

You will need [[https://lftp.yar.ru/][LFTP]] installed on your system. *It is already installed on TACC* If you are on a different system, you can install it using Conda.

Open the file =~/.lftp/rc= in a text editor. (You may need to first create the =~/.lftp= folder if it does not exist. Strictly speaking, this step is optional. You could run these commands every time you enter the LFTP shell to temporarily set the options.)

Add these lines to the =~/.lftp/rc= file.
<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
set ftps:initial-prot ""
set ftp:ssl-force true
set ftp:ssl-protect-data true
</verbatim></div>

Now you can use these commands to connect to Box (The second one is run inside the LFTP shell and will prompt you for the password you created.) The user name is the one listed in the account tab of the account settings page on UTBOX.

<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
lftp ftps://ftp.box.com:990
lftp ftp.box.com:~> user YYY@XXX.utexas.edu
</verbatim></div>

LFTP commands are a dialect of FTP commands. If you are not familiar with these, here is a [[https://lftp.yar.ru/lftp-man.html][reference]]. 

Here are some useful commands for working with your local directory from within the LFTP shell:
   * =lcd= lets you change local directories.
   * =!ls= lets you list the contents of your local directory.

To recursively copy directories *TO BOX*, you will need this command:

<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
mirror -R {local path} {remote path}
</verbatim></div>

This *does not* delete a file from ={remote dir}= if it has been deleted in ={local dir}= since the last backup. You would need to add the =--delete= flag for that type of sync, but be very careful when doing so! You can use the =--dry-run= option to see what would have been deleted before running it.

*Note:* Sometimes you may get an error looking like this during a =mirror -R= call. The file has been successfully transferred despite this message!
<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
mirror: Access failed: 550 Box: Item with the same name already exists (code=item_name_in_use, request_id=XXXXXXXXXXXXXXX). (FILENAME)
</verbatim></div>

*Note:* lftp may linked/compiled on some TACC servers such that you need an extra module loaded (i.e., Maverick2). Try this first if the above does not work.
<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
module load gcc 
</verbatim></div>

To recursively copy directories *FROM BOX*, you will need this command:

<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
mirror {remote_path} [local_path]
</verbatim></div>

[[https://www.cyberciti.biz/faq/lftp-mirror-example/][This website contains useful examples for both mirroring to and from a remote location]]

For copying all files in subdirectories (but not the directories themselves) from Box to TACC the following command will work:
<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
glob pget */*.gz
</verbatim></div>
Various changed to the glob pattern can be used to retrieve other files.

---++ Creating a Bookmark File

_This only works if you are UT-affiliated._ 

First, follow these instructions to [[https://support.box.com/hc/en-us/articles/360043694574-Box-SSO-Working-with-External-Passwords][set up an external password]].

Next, create a bookmark called <code>utbox</code>. 

_Security Note: this saves your external password in clear text in your home directory._

<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
lftp ftps://ftp.box.com:990
set bmk:save-passwords yes
user [your_EID]@eid.utexas.edu
[Enter Password]
bookmark add utbox
quit
</verbatim></div>

Then, you can log in using just one command!
<div style="border-color: grey; border-style: solid; border-width: 1px; padding:6px;"><verbatim>
lftp utbox
</verbatim></div>

Be sure to test your setup after logging in by running a command like `ls` because it will not give you an error if login failed!

It should list all of your main Box files.

This is also the standard way that we make it possible to mass download data as part of <i>[[https://github.com/barricklab/brefito][brefito]]</i> workflows.
This topic: Lab > ComputationList > CommandLineBox
Topic revision: r13 - 2024-05-18 - JeffreyBarrick