Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions _episodes/01.7-InteractiveComputingResources.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ Some resources (such as batch systems and `/pnfs/`) require "tokens" to access.
>It is considered quite rude to run large (> 2) numbers of analysis jobs on a shared gpvm. If you need to run more than 1 or 2, learn to use the [justin batch system](https://dunejustin.fnal.gov/docs).
{: .callout}

You can find out the load on the gpvms [here](https://fifemon.fnal.gov/monitor/d/000000004/experiment-overview?var-experiment=dune&orgId=1&var-pool=dune-global&var-pool=fifebatch&from=now-3h&to=now-5m)

### 2 build machines

Two specially configured interactive machines have been put in place explicitly to rapidly compile and build large software stacks. Nodes `dunebuild02` and `dunebuild03` have 90GB of memory and 24 CPUs each. These systems are for building code, not running or testing code.
Expand Down
262 changes: 151 additions & 111 deletions _episodes/02-storage-spaces.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion _episodes/02.3-cvmfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Users do use /cvmfs/ indirectly through the 'rcds' dropbox system that sends the

* CVMFS on DUNE wiki: [Access files in CVMFS](https://wiki.dunescience.org/wiki/DUNE_Computing/Access_files_in_CVMFS)

[Ifdh_commands]: https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands
[Ifdh_commands]: https://github.com/fnal-fife/ifdhc/wiki/Ifdh-commands
[xrootd-man-pages]: https://xrootd.slac.stanford.edu/docs.html
[Understanding-storage]: https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes
[DataCatalogDocs]: https://dune.github.io/DataCatalogDocs/index.html
Expand Down
2 changes: 1 addition & 1 deletion _episodes/03-data-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -725,7 +725,7 @@ When we are analyzing large numbers of files in a group of batch jobs, we use a
* [mrb reference guide (redmine)](https://cdcvs.fnal.gov/redmine/projects/mrb/wiki/MrbRefereceGuide)
* CVMFS on DUNE wiki: [Access files in CVMFS](https://wiki.dunescience.org/wiki/DUNE_Computing/Access_files_in_CVMFS)

[Ifdh_commands]: https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands
[Ifdh_commands]: https://github.com/fnal-fife/ifdhc/wiki/Ifdh-commands
[xrootd-man-pages]: https://xrootd.slac.stanford.edu/docs.html
[Understanding-storage]: https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes
[useful-samweb]: https://wiki.dunescience.org/wiki/Useful_ProtoDUNE_samweb_parameters
Expand Down
6 changes: 3 additions & 3 deletions _extras/sites.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ title: Lots of site links
- [Understanding Storage Volumes](https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes)
- [persistent usage](https://fifemon.fnal.gov/monitor/d/000000175/dcache-persistent-usage-by-vo?orgId=1&var-VO=dune) (bottom left):
- [DUNE Rucio Storage](https://dune-os.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/70a0baa0-8a3b-11ef-9dc8-5d2d451cf204?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!t,title:'Rucio%20Overview',viewMode:view))
- [ifdh commands](https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands)
- [ifdh commands](https://github.com/fnal-fife/ifdhc/wiki/Ifdh-commands)
- [XRootD Man pages](https://xrootd.slac.stanford.edu/docs.html)
- [ifdh commands (redmine)](https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands)
- [ifdh commands (redmine)](https://github.com/fnal-fife/ifdhc/wiki/Ifdh-commands)
- [Understanding storage volumes (redmine)](https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes)
- [How DUNE storage works:](https://dune-data.fnal.gov/tutorial/howitworks.pdf)
- [cvmfs](https://wiki.dunescience.org/wiki/DUNE_Computing/Access_files_in_CVMFS)
Expand Down Expand Up @@ -57,7 +57,7 @@ title: Lots of site links
- [Pre-2024 Official dataset definitions: dune-data.fnal.gov](https://dune-data.fnal.gov)
- [mrb reference guide (redmine)](https://cdcvs.fnal.gov/redmine/projects/mrb/wiki/MrbRefereceGuide)
- [CVMFS on DUNE wiki: Access files in CVMFS](https://wiki.dunescience.org/wiki/DUNE_Computing/Access_files_in_CVMFS)
- [Ifdh_commands](https://cdcvs.fnal.gov/redmine/projects/ifdhc/wiki/Ifdh_commands)
- [Ifdh_commands](https://github.com/fnal-fife/ifdhc/wiki/Ifdh-commands)
- [xrootd-man-pages](https://xrootd.slac.stanford.edu/docs.html)
- [Understanding-storage](https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes)
- [useful-samweb](https://wiki.dunescience.org/wiki/)Useful_ProtoDUNE_samweb_parameters
Expand Down
22 changes: 11 additions & 11 deletions _includes/02-storage-spaces.toc.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,32 +6,32 @@
- [Workshop Storage Spaces Video from December 2024](#workshop-storage-spaces-video-from-december-2024)
- [Introduction](#introduction)
- [Vocabulary](#vocabulary)
- [Interactive storage volumes (mounted on dunegpvmXX.fnal.gov or lxplus.cern.ch)](#interactive-storage-volumes-(mounted-on-dunegpvmxx.fnal.gov-or-lxplus.cern.ch))
- [Interactive storage volumes - mounted on dunegpvmXX.fnal.gov or lxplus.cern.ch](#interactive-storage-volumes---mounted-on-dunegpvmxx.fnal.gov-or-lxplus.cern.ch)
- [Your home area](#your-home-area)
- [at Fermilab](#at-fermilab)
- [at CERN](#at-cern)
- [Locally mounted volumes](#locally-mounted-volumes)
- [Network Attached Storage (NAS)](#network-attached-storage-(nas))
- [At Fermilab](#at-fermilab)
- [At CERN](#at-cern)
- [Grid-accessible storage volumes](#grid-accessible-storage-volumes)
- [Grid-accessible storage volumes - dCache](#grid-accessible-storage-volumes---dcache)
- [Persistent dCache](#persistent-dcache)
- [Scratch dCache](#scratch-dcache)
- [Tape-backed dCache](#tape-backed-dcache)
- [Rucio Storage Elements](#rucio-storage-elements)
- [CVMFS](#cvmfs)
- [What is my quota?](#what-is-my-quota)
- [Your home area at FNAL](#your-home-area-at-fnal)
- [Your home area at CERN](#your-home-area-at-cern)
- [The /app/ and /data/ areas at FNAL](#the-/app/-and-/data/-areas-at-fnal)
- [EOS at CERN](#eos-at-cern)
- [Fermilab dCache](#fermilab-dcache)
- [Summary on storage spaces](#summary-on-storage-spaces)
- [Monitoring and Usage](#monitoring-and-usage)
- [Commands and tools](#commands-and-tools)
- [The df command](#the-df-command)
- [Global data access tools](#global-data-access-tools)
- [authentication for dCache](#authentication-for-dcache)
- [ifdh](#ifdh)
- [example of an ifdh copy](#example-of-an-ifdh-copy)
- [xrootd](#xrootd)
- [What is the right xroot path for a file.](#what-is-the-right-xroot-path-for-a-file.)
- [The df command](#the-df-command)
- [xrootd or ifdh?](#xrootd-or-ifdh)
- [a test](#a-test)
- [What is the right xroot path - URI - for a file?](#what-is-the-right-xroot-path---uri---for-a-file)
- [Use metacat/rucio to find a file replica](#use-metacat/rucio-to-find-a-file-replica)
- [if file is in /pnfs/ you can directly convert its path to a URI](#if-file-is-in-/pnfs/-you-can-directly-convert-its-path-to-a-uri)
- [Quiz](#quiz)
- [Useful links to bookmark](#useful-links-to-bookmark)
Binary file added fig/Storage2026.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions gitadd.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
git add *.md
git add _episodes/*.md
git add code/*
git add fig/Storage*.png
#git add _episodes/01-introduction.md
#git add _episodes/02-storage-spaces.md
#git add _episodes/03-data-management.md
Expand Down
Loading