Bash while read line without subshell and arrays with IFS

Just a few notes about script where writing it took me far more time than it shoud :)

First, while read line without subshell. So I was attempting to do something like this:

count=0
export IFS=$'/n'
cat file.log | while read line; do
  let count+=1
done
echo $count

But because of the pipe, while runs in a subshell, so whatever I do to the count variable there is lost once I exit the loop, so echo prints 0 at the end. Using answer here I used this:

count=0
trap 'rm -rf $TMPFIFODIR' EXIT
TMPFIFODIR=$( mktemp -d )
mkfifo $TMPFIFODIR/mypipe
cat file.log > $TMPFIFODIR/mypipe &
export IFS=$'/n'
while read line; do
  let count+=1
done < $TMPFIFODIR/mypipe
echo $count

Now the second problem. Inside the loop I was using this to get first number from the line (because I think it is bit cheaper to do it this way than using sed or something):

numbers=( ${line//[!0-9]/ } )
count="${numbers[0]}"

This code did not worked for me and I was getting string with numbers and spaces in count - something that converting to array (these brackets) should take care about. It took me quite a bit of time to realize that when I'm exporting IFS to only newline, I'm breaking this. So I ended with just setting IFS for that read like this:

while IFS=$'\n' read line; do
  ...
done < $TMPFIFODIR/mypipe

Troubles pasting ritch-text content from CLI to Confluence

Using some custom script I'm generating some markdown (because it is easy to write) content, maybe it is status report or something. fOR EXAMPLE this:

$ cat /tmp/report.md
# Monday
* Watching cat videos
* Fixing what I broke at Fry

Now you want to paste it to, say, Google Document. First step is to convert to HTML:

$ cat /tmp/report.md | multimarkdown
<h1 id="monday">Monday</h1>

<ul>
<li>Watching cat videos</li>
<li>Fixing what I broke at Fry</li>
</ul>

Or using Pandoc:

$ cat /tmp/report.md | pandoc --from=markdown --to=html
<h1 id="monday">Monday</h1>
<ul>
<li>Watching cat videos</li>
<li>Fixing what I broke at Fry</li>
</ul>

Now to transfer it, very convinient way I was using is to copy it to the clipboard (and then just paste in in the editor with Ctrl+V):

cat /tmp/report.md | multimarkdown | xclip -sel clip -t "text/html"

When I needed to paste into Altasian Confluence WYSIWYG editor, it did not worked for me for some reason - only plain text was copied and the formatting was lost. But copying from normal web page worked. What is the difference? Thanks to this great answer, I have examined how clipboard looks like when I copy snippet from a web browser and noticed this:

$ # Selected and copied something from the web browser
$ xclip -o -selection clipboard -t TARGETS
TIMESTAMP
TARGETS
MULTIPLE
SAVE_TARGETS
text/html
text/_moz_htmlcontext
text/_moz_htmlinfo
UTF8_STRING
COMPOUND_TEXT
TEXT
STRING
text/plain;charset=utf-8
text/plain
text/x-moz-url-priv
$ xclip -o -selection clipboard -t text/html
<meta http-equiv="content-type" content="text/html; charset=utf-8"><ol>
<li>copy something from you web browser</li>
<li>investigate available types</li>
</ol>

So looks like I need that <meta http-equiv="content-type" content="text/html; charset=utf-8"><ol> string, so let's add it:

$ (echo '<meta http-equiv="content-type" content="text/html; charset=utf-8">'; cat /tmp/report.md | multimarkdown) | xclip -sel clip -t "text/html"

And here we go, pasting to Conflence works now!

Note: For some reason I do not understand, it seems to work the old way now once I pasted the new way for a first time :-/ Maybe above is not needed at all, YMMW.

Local variable in bash

Just a quick explanation on how local works in Bash I have been sending to somebody. Have this code:

$ cat /tmp/aaa 
function without_local() {
    variable1='hello'
    echo "Function without_local: $variable1"
}
function with_local() {
    local variable2='world'
    echo "Function with_local: $variable2"
}

echo "(1) variable1='$variable1'; variable2='$variable2'"
without_local
echo "(2) variable1='$variable1'; variable2='$variable2'"
with_local
echo "(3) variable1='$variable1'; variable2='$variable2'"

And run it and notice that variable1 behaves as global, variable2 wont leave function's context:

$ bash /tmp/aaa 
(1) variable1=''; variable2=''
Function without_local: hello
(2) variable1='hello'; variable2=''
Function with_local: world
(3) variable1='hello'; variable2=''

Difference in bash's $@ and $* and how it is expanded

I keep forgetting about this and I'm always confused what is happening but it is not that difficult. Example:

$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $@; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" $*; }
$ measurement_add "Hello world" 1
['Hello', 'world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$@"; }
$ measurement_add "Hello world" 1
['Hello world', '1']
$ function measurement_add() {     python -c "import sys; print sys.argv[1:]" "$*"; }
$ measurement_add "Hello world" 1
['Hello world 1']

Looking into man bash into Special Parameters section:

       *      Expands to the positional parameters, starting from  one.   When
              the  expansion  is  not  within  double  quotes, each positional
              parameter expands to a separate word.  In contexts where  it  is
              performed, those words are subject to further word splitting and
              pathname expansion.  When the  expansion  occurs  within  double
              quotes,  it  expands  to  a  single  word with the value of each
              parameter separated by the first character of  the  IFS  special
              variable.   That  is, "$*" is equivalent to "$1c$2c...", where c
              is the first character of the value of the IFS variable.  If IFS
              is  unset,  the  parameters  are separated by spaces.  If IFS is
              null, the parameters are joined without intervening separators.
       @      Expands to the positional parameters, starting from  one.   When
              the  expansion  occurs  within  double  quotes,  each  parameter
              expands to a separate word.  That is, "$@" is equivalent to "$1"
              "$2"  ...   If the double-quoted expansion occurs within a word,
              the expansion of the first parameter is joined with  the  begin‐
              ning  part  of  the original word, and the expansion of the last
              parameter is joined with the last part  of  the  original  word.
              When  there  are no positional parameters, "$@" and $@ expand to
              nothing (i.e., they are removed).

Remember that in Bash each command in pipeline is executed in a subshell

This took me some time to debug recently so wanted to share. Not new or so at all, but good to be reminded that from time to time :-)

$ function aaa() {
>   return 10 | true
> }
$ aaa
$ echo $?
0

Above can surprise a bit (why the heck I'm not getting exit code 10 when that return was executed?) especially when hidden in some bigger chunk of code, but with set -o pipefail I get what I wanted:

$ set -o pipefail
$ aaa
$ echo $?
10

Which processes have most open files and consumes most memmory?

For some testing, I wanted to watch number of open files by process and memory consumed (1) by all the processes (2) of same name to get some global overview. Graphing this over time is another exercise which can show trends.

E.g. following number of open files (includes all libraries loaded by a binary, opened sockets...) comes from freshly installed Spacewalk server from last evening and it is not surprising IMO:

# lsof | cut -d ' ' -f 1 | sort | uniq -c | sort -n | tail
    121 cobblerd
    121 sshd
    122 gdbus
    131 master
    264 gssproxy
    282 gmain
    344 tuned
   1256 httpd
   4390 postgres
  25432 java

And this is total memory per processes with same name from same server - again nothing unexpected:

# ps --no-headers -eo rss,comm >a; for comm in $( sed 's/^\s*[0-9]\+\s*\(.*\)$/\1/' a | sort -u ); do size=$( grep "\s$comm" a | sed 's/^\s*\([0-9]\+\)\s*.*$/\1/' | paste -sd+ - | bc ); echo "$size $comm"; done | sort -n | tail
16220 tuned
18104 beah-fwd-backen
18664 beah-srv
23544 firewalld
24432 cobblerd
26088 systemd
26176 beah-beaker-bac
71760 httpd
227900 postgres
1077956 java

BTW man ps says following about RSS (which is used above):

resident set size, the non-swapped physical memory that a task has used (in kiloBytes).