Skip navigation

Category Archives: unix

Escaping a forwardslash in awk like

awk  '/javascript\//'

doesn’t work
The way to do so is using the hexadecimal value of “/” which you can figure with the “od” unix command:

echo "/" | od -x

which returns:
0000000 0a2f

then doing:

awk '/javascripta2f/'

works fine!

Due to a disk failure on a server I needed to preserve several databases created for students. Some of them were empty, consecuently doing:
mysqldump with the option --all-databases yielded an error and didn’t work.

What worked instead, was iterating through and mysqldumping them individually with this shell oneliner:

for I in $(mysql -uroot -p****** -e 'show databases'); do  mysqldump -uroot -p****** $I > $I.sql; done;

I compressed those created files into one tar inmediately after:

ls -1 | xargs tar -cvzf backupOfDatabases.tgz

I needed to inspect a relatively small portion of a large log file (~1Gb), which will make chock even powerfull text-editors like vi(m) or emacs

I proceded in two steps:
1) found the match in the file and pulled the line number

awk ‘/ May 10 /{a=$0; b = NR;}END{print a,” :: “,b}’ log.txt
which yielded:
Thu May 10 02:17:05 ART 2012 :: 29199076

2) then I dumped the content from the that line and filtered it with head
tail -n +29199076 log.txt | head -n 100
That is possible with the trick of using “tail -n +(N)” which brings lines from the N line onwards

As and alternative to the last one, as explained here, sed could’ve been used in the following manner:
sed -n -e 29199076,29199176 -e 29199077q log.txt
(the last parameter, for efficiency, tells to quit at the limit line + 1 )

I have been enjoying the window placement hotkeys in Ubuntu ever since I accidentally discover them. The possibility to center/maximize or reposition windows to any side of the screen becomes essential when you tasted the power of mouseless interactions. I only wished to find something similar for my cygwinized laptop, and was happy to learn that Windows 7 alows some window management via the keyboard with the following shortcuts:
:: windows-key + up-arrow: maximizes the current window
:: windows-key + (left/right)-arrow: docks the current window in the left or right side of the screen
:: windows-key + down-arrow: minimizes, or restores the size of a maximized window

Writing heredocs in the shell from emacs presented to me a little nuisance: since the RETurn key is bound to ‘comint-send-input -a function that sends the text to the shell process- I coudn’t re-edit the previous line after having pressed it to insert a simple line break. (I should add that this is actually the usual unix bash behaviour, I only happen to use bash from emacs and so wanted to overcome the problem there)

Yesterday it dawned on me to just use “C-q C-j” under that situation, (that’s how you stick in a line feed in emacs regardless of the mode you are in).

Given that the perspective of typing “C-q C-j” several times looked rather awkward I thought of binding RET with the control key for that:

(global-set-key (quote [(control return)]) (quote newline))

But then, the idea of going the other way around showed more appealing: why not to make newlines with a simple RET, and use the Control key to send the input? Besides, it makes sense to prevent hitting return by accident, I hate when my pinky lands there when searching the pipe “|” key.

While at it I decided to also rebind the TAB functionality so I could further improve the readability of my scripts by indenting them at the same time.

Implementing those changes is simple when you’ve got the luxury of a highly-configurable tool:

(defun my-shell-hook ()
  (local-set-key (quote [(return)]) (quote newline))
  (local-set-key (quote [(control return)]) (quote comint-send-input))
  (local-set-key (kbd "M-i")  'my-unindent)
  (local-set-key (kbd "C-i")  'indent-or-complete))

(add-hook 'shell-mode-hook 'my-shell-hook)

UPDATE: After a day trying this set-up I think that I’d rather stick to the normal binding of the RET key. I’m leaning toward using C-j and C-m instead, as I caught myself reaching to either of them to insert carriage returns. Here are the modified bindings in replacement:

(defun my-shell-hook ()
  (local-set-key (kbd "M-i")  'my-unindent)
  (local-set-key (kbd "C-i")  'indent-or-complete)
; (local-set-key (quote [(return)]) (quote newline))
; (local-set-key (quote [(control return)]) (quote comint-send-input))
;; I actually liked these two better for adding  newlines, leaving the return with it's default behavior
  (local-set-key (quote [(control j)]) (quote newline))
  (local-set-key (quote [(control m)]) (quote newline))
  (local-set-key (quote [(return)]) (quote comint-send-input)))

(add-hook 'shell-mode-hook 'my-shell-hook)

Now that I’m getting fond of using a heredoc to insert mysql scripts into the bash shell, have put this into my emacs initialization file. The same shortcut (Control+Shift+f11) will conveniently write different parameters depending in which shell I’m sitting in.

UPDATE: seems that I spoke too soon. Asking about the present working directory, formerly “(let ((a (shell-command-to-string “pwd | tr -d ‘\\n'”)))”, didn’t actually work when having opened shells of different servers via Tramp.
The correction below (less elegant) depends on the shell buffer being created/renamed with an identifiable name -which could be the hostname or whatever consistent nomenclature we choose-.
Anyway, until figuring a better way, this does the job of inserting the right parameters into each shell:

(defun my-heredoc-sql-invocation-from-shell ()
"Insert the appropiate parameters to run a heredoc mysql query depending on which shell I'm in"
  (let ((a (buffer-name (window-buffer (minibuffer-selected-window))))
        (b nil))
    (cond ((string-match "serverA" a) 
           (setq b "mysql -uroot mainDbToHitAt_A--password=`cat /etc/security/mysqlpassword` -t -vv <<\\!"))
          ((string-match "serverB" a) 
           (setq b "mysql -ualpha mainDbToHitAt_B --password=`cat /etc/security/mysqlpassword` -t -vv <<\\!"))
          ((string-match "serverC" a) 
           (setq b "mysql -uroot mainDbToHitAt_C  -t <<\\!")))
    (insert b)))

;; key shortcut to bind it to
(global-set-key (kbd "C-S-<f11>")  'my-heredoc-sql-invocation-from-shell)

here’s an example of how to run php in the command line, something I almost never do and would like to remember by writing it down.
Simply echo the php statement (within its corresponding tags) and pipe it to the php service. Eg:

echo ‘<?php $str = ‘2144338’; echo substr($str,0,2); ?>’ | php

or even more usefully:

echo ‘<?php  phpinfo(); ?>’ | php

I sometimes forget about the exact name of a file I was working a while ago. Luckily the way to state a range of time with find is quite intuitive; say that we want to see the files modified in a lapse of 14 days starting from a week ago:

find . -ctime +7  -ctime -21 # (older than 7 days but newer than three weeks ago)

There are simply 3 options of numeric arguments for time

  1. +n –> More than ‘n’ days ago.
  2. -n –> Less than ‘n’ days ago.
  3. n —> Exactly ‘n’ days ago.

I struggled the other day doing some sys admin work for recovering data from a single table of our database. Editing big files (of about several gigabites) is no-picnic even for vi(m) or emacs, so it wasn’t trivial to find a quick way to isolate the parts needed. For what is worth here’s the method I’ve followed with success resourcing to simple cat and sed commands in my command line:

  1. Get the creation statement for the table to be recovered

    cat your_entire_backup_file.sql | sed -ne '/Table structure for table `your_table`/,/-- Dump/p' > table_creation.sql
  2. Get the data

    cat your_entire_backup_file.sql | sed -ne '/INSERT INTO `your_table`/,/Table structure/p' > data_dump.sql
  3. Join the two into a single file

    cat table_creation.sql data_dump.sql > data_for_single_table_to_copy
  4. Optionally, in case you need to extract some rows only from that previous instance of the table, as it was my case with records deleted by mistake, you might want to create a temporary table from where to later perfom the selection of the desired rows. In order to do that, the table name should be altered from the creation and insertion statements:

    sed -i 's#your_table#your_temp_table#g' data_for_single_table_to_copy.sql
  5. Now we are ready to create that temporary table with its data inside our database:

    mysql -u 'your_username -p your_database_name < 'your_path_to_the_file/data_for_single_table_to_copy.sql
  6. Voila!, the table is there containing the information you needed. Now is up to you to extract and reinsert whatever you wanted inside the original table

Note: See that different parameters could be used to isolate and put together table creation and data parts in only one pass. Also the awk command might be used instead since, like sed, it permits collecting portions by matching from the beginning to the ending block of text. Just make sure you know the order of the table after the one you are picking.

awk '/Table structure for table `your_table`/,/Table structure for table `your_next_table`/{print}' your_entire_backup_file.sql > data_for_single_table_to_copy.sql

In case the table to extract happens to be the last one (which again, you could just know with a mysq “show tables” command, modify the last part of the regexp to match it accordingly.

Having discovered the command cpio (wich acts like tar but preserving symbolic links) to recursively put into a single file all the directory structure and files of an entire website, I started to explore its use along with other ways to securely move files between hosts. The command rsync seems like an extraordinary find in that regard.

Here are some neat examples for backing up (or mirroring) sites with ssh I don’t want to miss:

There’s a really easy way to make a remote directory be an exact match of a local directory via an ssh connection. And that’s to use rsync to do the job. It’s smart and only sends enough data to sync the remote dir to the local one. Assuming that you have a local dir named master-copy and the remote system’s repository is in a directory named slave-copy you’d do:

rsync --rsh=ssh --delete -Cav master-copy/* user-name@remote.dom.tld:/path-to/slave-copy

That command will recursively sync everything found in master-copy to the remote. Any files found on the remote that aren’t in master-copy or one of its sub-dirs will be deleted from master-copy. Only changed or new files will be transferred to the remote.

to which another expert adds:

The rsync idea is good — but I for my part always try NOT to install any additional software not essentially required. Today, ssh is “a must” on most systems and you can go with the cpio approach I’ve mentioned.
I must admit the rsync thing is transferring less data if stuff is already on the target side, but this is not the case when running
source=/var/tmp ; target=/var/backup
cd $source ; find . -mtime -1 -type f -print | cpio -ocB | ssh -l username host-b "cd $target ; cpio -icBduml"


Another suggestion to copy the entire directory tree at /some/path to via SSH is to do:

find /some/path -print | sort | cpio -o -Hnewc | ssh -C "cpio -idvum"

Finally, there’s more to chew with this thorough example on how to mirror a site with rsync