Physics 230 -- Lab 8 (Data Processing)

Fall 2015 - Transtrum

Labs developed by John S. Colton, BYU Physics & Astronomy, based in part on past labs by Branton Cambell and Steve Turley

Most scientists spend a great deal of time working with data.  In this lab, we will introduce some important tools for importing and exporting a variety of data types to and from Mathematica, and will also practice the art of manipulating complicated multidimensional lists (i.e. arrays) of data.

Clear["`*" ]
SetDirectory[NotebookDirectory[]]    (* this command is explained below, but given here for your reference *)

Units, Packages, System Variables

Units

Important Note: There were apparently some major changes in the way Mathematica handles units going from Mathematica 9 to Mathematica 10. The following section was written for Mathematica 10. If you are using Mathematica 9, some things will likely not function in quite the same way.

One useful Mathematica feature for physics students is the ability to use and calculate with physical units. See for example, this Units Overview Tutorial. Values with a magnitude and dimension can be defined using the Quantity command.

position = Quantity[12,"Meters"]
time = Quantity[5,"Seconds"]
vel = position/time //N

You can also add units via control-equals, which activates a connection to Wolfram Alpha and allows for a bit more flexibility in spelling of units. I myself find that it’s the easiest way to add units. To get the odd looking thing in the next box, I typed “length1=” then “control-equals” (to activate Wolfram Alpha parsing), then “14.3 km”.

length1=LinguisticAssistant

Try typing something with units via a control-equals on your own, so you can see how it works. For example, type “length2=” then “control-equals” then “14.3 kilomeaters”  (a deliberate misspelling which Wolfram Alpha should be able to handle).

(* empty box for you to use *)

Many physical constants (with their units) are built into Mathematica. For example:

h=Quantity["PlanckConstant"]
h=LinguisticAssistant   (* same thing, using control-equals and "plank's constant" after the equals sign *)
k=Quantity["BoltzmannConstant"]
k=LinguisticAssistant (* same thing, using control-equals and "boltzmann's constant" after the equals sign *)
c=Quantity["SpeedOfLight"]
c=LinguisticAssistant (* same thing, using control-equals and "speed of light" after the equals sign *)

The UnitConvert function allows you to convert units:

UnitConvert[length1,"Cubits"] //N
UnitConvert[c,"Feet"/"Nanoseconds"] //N   (* Notice the "divided by" symbol as part of the argument.*)  (* and notice that the speed of light is almost exactly 1 foot per nanosecond *)

Mathematica can do more with units than just simple conversion. Here’s an example where I’m calculating (a) the current gravitational force between the earth and the moon, and (b) the amount of energy it would take to move the moon 1000x farther away from the earth (using the work from point a to point b=(∫_a^b)Fdx).

(* part a *)
G =LinguisticAssistant ;
me=AstronomicalData["Earth", "Mass"]  ;
mm = AstronomicalData["Moon", "Mass"]  ;
startingdistance=LinguisticAssistant ; (* to get this line I typed "startingdistance =" <control-equals> "distance from earth to the moon" *)
currentforce = G me mm/startingdistance^2 ;
UnitConvert[%, "Newtons"]
(* part b *)
force[x_] =  -G me mm/x^2;
energytomovemoon = -Integrate[force[x],{x,startingdistance,1000 startingdistance}]; (* negative sign because the force to move the earth is opposing the force of gravity *)
UnitConvert[energytomovemoon, "Joules"]

Built-in functions

As you know, Mathematica has many, many, built-in functions. You can use the command Names to see them. There are thousands of built-in functions, so let’s just look at all of the Mathematica functions and symbols that start with a "W".

Names["W*"]

Here’s a way to see how many built-in functions there are. (Any functions or variables you’ve defined yourself will be included in the number.)

startupnames = Names["*"];
Length[startupnames]

Packages

Packages are a way to enhance Mathematica’s capability by loading in additional commands for you to use. Loading a package is done with the Get command (shortcut <<), or with the very similar  Needs command, with a  `  after the package name to signify that the package is to be loaded into the current session:  

<<PackageName`

or 

Needs[“PackageName`”]

The Standard Extra Packages available in Mathematica 9.0 are listed in the listed in the Documentation Center. Typically you may run across a package when searching Help for a specific command or topic. For example, I discovered that Mathematica has built-in commands for working with resonance lines in atomic spectra. This is the Resonance Absorption Lines Package. The first input line of the help file says:  

<<ResonanceAbsorptionLines`

That’s a giveaway that the things discussed in this particular help file are not part of the standard Mathematica functionality, but rather new functions will become available after you load the relevant package (the “Resonance Absorption Lines” package, in this case). Evaluate the cell below to see the new functions and symbols defined after loading the ResonanceAbsorptionLines package. (Note the neat trick using Complement to see which commands currently loaded were not part of the startupnames variable defined above.)

<<ResonanceAbsorptionLines`
currentnames = Names["*"];
Length[currentnames]  (* includes additional functions loaded via the package *)

newnames = Complement[currentnames,startupnames]
Length[newnames]  (* how many new ones there are *)

Using this package we can plot absorption lines for different elements, or investigate a range of wavelengths. The sodium ion lines around 5890 Angstroms are used in old fashioned street lights.

ElementAbsorptionMap[H]

WavelengthAbsorptionMap[5000 Angstrom, 7000Angstrom]

System variables

Names beginning with the "$" character are called system variables.  Mathematica relies on their values to perform important tasks that are normally invisible to the user. We’ve already seen one in an earlier lab, the $Assumptions variable. Usually you won’t need to modify these, but situations might arise where you would want to.

Names["$*"]  (* here's a list of all the system variables *)

To see the current value of a system variable, just type in the variable name.

$ImportFormats   (* the available formats for the Import command *)

Directory Navigation

In order to read in data from files, Mathematica has to know details about where the files exist. Same thing if you want to save data to a file. This is typically done by specifying a “working directory”, i.e. the location where all file reads/writes are steered. Some basic information is given in the tutorial on Naming and Finding Files. You can find more advanced info in the File Operations guide and the links found there.

These are two especially important directories:

$InitialDirectory   (* The initial working directory *)
Directory[]  (* The current working directory *)

You can set the working directory to be anywhere you like on the computer--just be careful, as the backslash character needs to be entered as a forward slash (or two backslash characters together also seem to work. Nice of Mathematica’s help file to mention either of those... NOT).

(* first change the directory location to one that exists on your computer *)
SetDirectory["/Users/dave/projects"]  ; 
Directory[]
SetDirectory["../"]     (* To go "up" a directory *)

Here we set the working directory back to the default $InitialDirectory and use FileNames to view the files in the current directory.

SetDirectory[$InitialDirectory];
FileNames[]

It would make a lot of sense if when you start Mathematica by double-clicking a notebook file, the working directory would automatically be set to the directory containing that file. However, that isn’t the case. Therefore here is a helpful shortcut to do that for you. I recommend you use it at the top of all of your workbooks from now on.

SetDirectory[NotebookDirectory[]]   (* This is EXTREMELY helpful!!! I use it all the time!! *)

File Copy & Delete

Mathematica can copy and delete files using the operating system on your computer. To demonstrate that, first evaluate this cell to create a file in your working directory called "oldtest.txt".  Check to make sure that it was created.

Export["oldtest.txt",{1,2,3,4,5}]

Here’s how you would copy "oldtest.txt" to a new file called "newtest.txt".  Check to make sure that it was copied.

CopyFile["oldtest.txt","newtest.txt"]

Here’s how to delete both files.  Check to make sure that they were deleted.

DeleteFile[{"oldtest.txt","newtest.txt"}]

Data Import/Export/Processing

Import/export of expressions

Importing and exporting complete Mathematica expressions is really easy. You can browse the tutorial page on Reading and Writing Wolfram System Files for lots of information.

The bottom line is that just as Packages can be imported via the  << (Get) command, so can your own Mathematica expressions be imported via  <<  and exported via  >> (Put). The shortcuts are very intuitive, as can hopefully be seen by these examples.

Expand[(a+b)^10]
% >> "test.txt"    (*  this stores the expression in a text file called test.txt *)

You should manually open the “text.txt” file in a text editor such as Notepad or Notepad++ (a great free program which is installed on all departmental lab computers) to see what the saved expression looks like. You can also view the contents of the file via Mathematica's FilePrint function:

FilePrint["test.txt"]

To read the information back in, you just need to use  <<.

expansion = <<"test.txt"

Importing numerical data

Mathematica supports many formats for Importing and Exporting data. In this section we’ll talk about numerical data. Later in this lab we’ll get to some other data types such as images and audio.

Import review.  We’ve already used the Import function once to grab data from a file, back in Lab 2 when we were discussing lists. Let’s revisit that. Download the file Th000088.txt again (it contains reflectance data) and put it into the directory where you have saved this notebook.

Before proceeding with these next commands, which will create an array called “tdata” and fill it with the data, look at the data file itself with a text editor such as Notepad++. It’s a very common type of data file--a row of header information, followed by the raw data. Each row of data begins with one independent variable value (wavelength in nm), then has several dependent variables. The numbers within each row are separated by tabs.

OK, here’s the command used in Lab 2 to read in the data. Notice the “Table” option that is used. That tells Mathematica that it will be dealing with a “tab-delimited” text file (i.e. numbers separated by tabs).

SetDirectory[NotebookDirectory[]]
tdata = Import["Th000088.txt","Table"]

The Lab 2 exercise involved you manipulating the list--you needed to drop the first row of data, extract the wavelength & reflectance data as pairs of numbers, then plot them using ListLogPlot. Here are the commands I myself used to accomplish that. Make sure you understand how/why all of my commands work (your own commands for that assignment likely did things slightly differently).

ourdata=Drop[tdata,1];   (* I've defined "ourdata" so I can refer to it below. It's not needed for this task. *)
Transpose[%];  
Drop[%,-2];
xydata = Transpose[%]
ListLogPlot[xydata]

A trick worth knowing.  In my commands above, the first Transpose followed by the Drop[%,-2] command served to drop the final two columns. Probably most of you did something similar to that, but here’s another neat way of doing the same thing. First, observe what ourdata //Transpose looks like:

ourdata //Transpose   (* it's a collection of four separate lists *)

Here’s the trick:

{wavelength, reflect, garbage1, garbage2} =  ourdata  //Transpose ;     (* look at the left hand side of the equals... this is slick! *)
newxydata= {wavelength, reflect} //Transpose

Things to notice: 
ourdata // Transpose is a collection of four lists--two that we care about, and two that we don’t.  By writing the left hand side of the first command as  {wavelength, reflect, garbage1, garbage2} , I’ve automatically defined “wavelength” to be the first of those four and “reflect” to be the second. Then I can deal with them however I want. Defining distinct parts of arrays like that is very slick and can be very useful when reading in files.

Exporting numerical data

Let’s suppose now that you wish to re-save the data into a new file, but just the wavelength and reflectivity numbers that are now in the “xydata” array and not the header row or other two columns that were part of the original tdata array.

The Export command has a nice  “Table” format specifier just like the Import command had. The “Table” format uses tabs to delimit (i.e. separate) the columns in the output file, though options are available for using other delimiters such as commas or spaces.

Assignment 1. Exporting data

(a) View the xydata array in TableForm to see what it will look like in a file if you use the Table format specifier.
(b) Now Export the xydata array to a file in your working directory named "xydata.txt" using the Table format. (Note that if you were to use a filename with a ".dat" extension, "Table" would be the default format and would not need to be specified.)  View the xydata.txt file to make sure things worked as expected.
(c) Another very common input/output format is the csv or “comma separated variables” format. That format uses comma to separate the data on each line. Mathematica is smart enough that simply by giving the filename a “.csv” extension, it will automatically save the data with commas. Export your xydata array to a file called “xydata.csv” and use a text editor to verify that you now have commas where the xydata.txt file had tabs. (As an added bonus, csv files can typically be opened as spreadsheets in Microsoft Excel by simply double-clicking the file in Windows Explorer.)

Interpolating functions

Sometimes you may want the data you have imported to be able to be treated just like a regular function. For those times, the Interpolation function is handy. It allows you to take a list of values (e.g. {x,y} pairs) and define a function based on them. Then you can evaluate the function at any point and it will interplote a value based on the data you supplied. You can even use it in a regular Plot command instead of needing to do a ListPlot.

xyfunction=Interpolation[xydata]  (* Values between the data points are now interpolated. *)
xyfunction[200.5]  (* This point doesn't exist in the original data, but it exists now. *)
Plot[xyfunction[x],{x,172,250}]  (* Using Plot instead of ListPlot. *)

Formatting output

When you did Assignment 1 above, the output file  xydata.txt  probably had nice, clean columns of data inside of it. However, Mathematica sometimes has issues in this regard. Consider this new set of data which I’ve called “morexydata” (it’s a noisy parabola).

morexydata= Table[{n,n^2+5*RandomReal[{-1,1}]}, {n,0,10,0.2}];
% //TableForm

It looks like a nice, clean, table of data, so let’s save it to "morexydata.txt" using our standard method.

Export["morexydata.txt",morexydata,"Table"]

Now open up  morexydata.txt  with a textedit and look at what actually got saved.  Ugly!!   Not only is the second column adding a bunch of unwanted decimal places, the first column is also messed up, apparently because there’s a very small rounding error that occurs when the Table command generates the multiples of 0.2.

There are various ways you can deal with this type of situation. Here’s a method that involves converting numerical data into nicely formatted strings prior to export.

niceoutput = Map[ToString[NumberForm[#,{10,6},NumberPadding->{" ","0"}]]&,morexydata,{2}]
Export["morexydata2.txt",niceoutput,"Table"]

Open the morexydata2.txt file in an external text editor to verify the results. Much nicer!

Now change the values above in {10, 6} and {“ “, “0”} to see what they do.

Short Detour: Strings

String basics

That last example involved translating your number into a formatted string. A String is a variable type in Mathematica, just like an integer, real number, complex number, Boolean, etc. We’ve used strings before--they are just text characters grouped together via surrounding quotation marks--but we haven’t really stopped to talk about them. Just as numbers and Booleans have a lot of built-in functions for manipulating them, so do strings. We could probably spend an entire lab just on string functions, so we’ll have to stick to the basics here.

Numbers can be transformed into strings, and strings that look like numbers can be transformed into numbers. ToString and ToExpression are the commands.

thisisanumber = 12345
thisisanumber + 5
thisisastring=ToString[thisisanumber]  (* this is NOT a number. thisisastring is a string that LOOKS like a number *)
thisisastring +  5   (* addition doesn't work because thisisastring is not a number *)

anotherstring = "2468"  (* this is NOT a number *)
anotherstring + 5  (* this isn't going to work *)
anothernumber = ToExpression[anotherstring]  (* NOW this is a number *)
anothernumber + 5

The ToExpression command works on strings that look like any expressions (algebraic, etc.), not just on strings that look like numbers:

zstring = "-1 + 3 x + x^2"   (* this is a string, NOT an expression... *)
Plot[zstring,{x, -3,3}]  (* ... and so this will not work at all *)

zexpression = ToExpression[zstring]  (* this converts the "x^2 + 3*x - 1" string into a real expression... *)  
Plot[zexpression,{x, -3,3}]   (* ... and so this WILL work *)

In some sense, a string is like a list of single characters. Therefore a lot of things that we’ve done with lists can be done with strings using string versions of the commands: reversing, inserting, replacing, etc. Many of the string-specific functions can be found in these two guides: String Operations and String Manipulations.

Some quick demonstrations:

sentence = "This is " <> "a sentence."   (* <>  is shortcut notation for StringJoin, which acts like Append does for arrays *)
StringLength[sentence]   (* how many characters *)
StringReverse[sentence]
StringInsert[sentence, "very cool ",11]
StringReplace[sentence, "is" -> "zzz"]
StringPosition[%, "zzz"]  (* where are the zzz's? Gives first and last index for each found string *)
StringPosition[%%, "zzz", 1]  (* just the first zzz *)
Sort[{"This", "is", "a", "list"}]  (* Sort puts string array elements in alphabetical order *)

Merging numbers into a string

Often you may want to merge one or more numbers into a string, perhaps when (for example) displaying the result of a calculation. There are two fairly easy methods:
1. You can convert the numbers into strings, then use  <>   (StringJoin)
2. You can use StringForm, followed by a set of numbers. Each number that is to be merged into the string is given the symbol  ``  in the string as a temporary placeholder.

number1 = E^1.5
number2 = Log[1.5]
sentence1 = "The exponential was " <>  ToString[number1] <> " and the natural log was " <>  ToString[number2] <>"."
sentence2 = StringForm["The exponential was `` and the natural log was ``.", number1, number2]

StringForm can be combined with NumberForm to specify how many significant digits of the numbers should be displayed.

sentence2 = StringForm["The exponential was `` and the natural log was ``.",NumberForm[number1, 9], NumberForm[number2,8]  ]

Incidentally, the StringForm command works with other things too, not just numbers.

StringForm["Performing the integral `` results in ``.",HoldForm[Integrate[x^2,x]],Integrate[x^2,x]]
(* HoldForm means it shouldn't do the integral yet *)

Converting strings to lists

Although a lot of the list functions have string analogs, sometimes they don't. For example, I was once unable to find a string analog to the PadLeft list function when I wanted to display a list of integers with leading zeroes. I was able to do it by creating a function which converts a number to a string, then converts the string to a list of characters (using Characters), then applies PadLeft to the list, then does a StringJoin to convert the list of characters back into a single string.

fixedwidth[num_Integer]:= StringJoin[
PadLeft[Characters[ToString[num]],8,"0"]    (* adds array elements of 0's on left so array has 8 elements in it *)
]
Do[Print[fixedwidth[i]],{i,1,15}]

Assignment 2.  A string function

Create a function using Module which takes two arguments: an expression containing x and a numerical value to plug in for x, then prints out the phrase “The value of <expression> when x = <number> is: <the answer>”.

Import of string (text) data

In the previous section we imported some undesired numerical data as a string, so we could throw it out. However, sometimes you may be dealing with data that is actually in string (text) format. For those cases, the tutorial on Reading Textual Data may be helpful.

To set up this example, download the “funwithstring.txt” file from the course website into your working directory. Before running the commands in the next box, look at the file with a text editor so you know what you’re dealing with. Then use these commands to Import it into a list called firsttry (assuming a “Table” format) and examine it using TableForm.

firsttry= Import["funwithstring.txt","Table"];
firsttry//TableForm

The innocent-looking data has actually not imported well at all. The main issue is that some of the names got imported as last & first columns, whereas others got imported into last & first & middle initial. Therefore some rows have 8 elements and some have 9. And to add insult to injury, one of the street names was “North Hitchcock Way”, so , it got imported into three separate column elements instead of two columns like the other street names (because the table import assumed all elements were separated by spaces). 

To solve this problem, after I noticed that the data was all separated by commas I just manually renamed the file to “funwithstring.csv”. Do that now, yourself. Then you can use import with no additional specifiers, and no issues.

IMPORTANT: Depending on what options you have in place with Windows, you might not be able to change the file extension (the “.txt” or “.csv”). The default behavior of Windows is to not allow people to change file extensions, so if it looks like you renamed the file to “funwithstring.csv” but it still shows up as a text file, you undoubtedly really renamed the file to “funwithstring.csv.txt”. To change the default behavior in Windows 7, when in Windows Explorer press Alt-T to get to the Tools menu, then choose Folder Options, then select the View tab, then uncheck the box that says “Hide extensions for known file types”. Then you will be able to see and modify the file extension.

secondtry= Import["funwithstring.csv"];
secondtry//TableForm

As another way around the issue, you could also import the data using the {"Text","Lines"} specifier. That reads in each line as its own string, giving you an array of strings that represents the file contents.

thirdtry = Import["funwithstring.txt",{"Text","Lines"}]   
%//InputForm   (* InputForm shows you where the quotation marks are *)

(StringSplit[#,", "]&)/@thirdtry   (* manually splitting each string at the commas *)

% //TableForm  (* to see things in a nicely formatted table *)

Multimedia

Multimedia files involving images, audio and/or video, have become important methods to record, store, share, and visualize scienfic data. Mathematica supports a large number of Import/Export formats.  The Import and Export reference pages are fairly limited, but contain links to a number of good tutorials. You can also view the complete list of supported formats, grouped by media type or grouped alphabetically, to learn about the special options available to individual formats.  As in the “csv” examples above, if you give your filemame a common extension (such as “.jpg” in this context), Mathematica should automatically figure out how to deal with that particular format. Otherwise, you can provide a format specifier (e.g. "JPEG") as an argument to Import or Export.

Loading data from “Example Data”

Mathematica has a lot of miscellaneous data that can be used for testing purposes, called ExampleData. Here’s a list of all of the categories.

ExampleData[]

You can also list all of the data available within a particular category. Here’s all of the data in the “test image” category.

ExampleData["TestImage"]

To view/access one of those test images, you use the ExampleData function like this:

colorpeppers = ExampleData[{"TestImage","Peppers"}]

Loading data from internet

In addition to loading images that are in the ExampleData, one can also Import images from disk or from the internet, just like you would load numerical data files. For example, here’s a picture I took of the Nauvoo temple:

temple = Import["http://www.physics.byu.edu/faculty/colton/personal/lds/temple1.jpg"]

Images

The data involved in producing an image is stored as a 2D array of pixels. Each pixel can contain either one number or a triplet of numbers in a list. If a single number exists, Mathematica will interpret the data as a black and white image. If a triplet exists, it will be a color image, with the three numbers indicating how much red, green, and blue there is in the pixel. The Image command causes the pixel information to be displayed as a picture. Here are some examples, but just using random numbers so that the “picture” doesn’t actually look like anything.

bwarray=RandomReal[{0,1},{20,10}];  (* a 20x10 array of random numbers between 0 and 1 *)
%//MatrixForm

Image[bwarray]  (* a black & white (grayscale) image of the random data, 20 rows and 10 columns of pixels. 
You can enlarge the image by clicking on the image and then dragging on a corner. Make it big enough that you can see the individual pixels. *)

colorarray=RandomReal[{0,1},{20,10,3}];  (* a 20x10 array of pixels, each pixel now being a 3 element list *)
%//MatrixForm

Image[colorarray]  (* This now produces a color image, because Mathematica recognizes that there's a list of three numbers given for each pixel *)

You can access individual pixels using the regular list commands.

bwarray[[1,1]]  (* current value of upper left pixel *)
bwarray[[1,1]]=1  (* sets upper left pixel to pure white; click and drag the corner of image to enlarge it and verify that *)
Image[bwarray]

colorarray[[1,1]]  (* upper left pixel *)
colorarray[[1,1]]={1,0,0}  (* sets upper left pixel to pure red; enlarge image to see it *)
Image[colorarray]

Metadata. In addition to the numbers which make up the pixels, some “metadata” is also often stored in the image which can include information about how the numerical data is to be used, where the picture was taken, and so forth. You can see the metadata in the peppers image, for example, by removing the semicolon in the next cell, then evaluating it. There’s a dizzying amount of numbers, but then at the end of the numerical data you can see:  “Byte”, ColorSpace -> “RGB”, Interleaving -> True. That’s the metadata. (I recommend you delete the lengthy output when you are done looking at it so it doesn’t clutter up your notebook.)

colorpeppers//InputForm;  (* remove semicolon to see output *)

You can access the pixel information in a pure numerical list format via the ImageData command:

colorpeppersdata = colorpeppers//ImageData; 
colorpeppersdata[[1;;3]]  //TableForm  (* the first three rows of pixels *)

Assignment 3. Black and white peppers

The goal of this problem is to produce a black and white peppers image based on the color image of the peppers seen above. Part (a) is an easier version of the problem, then part (b) applies the same technique to the peppers image itself. 

(a) Working with your colorarray data from above, figure out how to replace each pixel, i.e. each three element list, with a single number that is the average of the three values. Use the Map command, which we often shortcut as  /@.  

Hint: You cannot actually easily use the shortcut notation in this case, because the default behavior of Map (and  /@ ) is to perform an operation on each row of the matrix. We need to perform an operation (i.e. average the numbers) on each pixel in the matrix. Using Map with a  {2}  for the “levelspec” argument will do this. (Look at the help file to see where that goes.) You will know that you have succeeded when you can view the result in MatrixForm, and see a 20x10 list of numbers (rather than a 20x10 list of triplets).  And when you have done that, you will be able to use Image to see a black and white version of your colorarray data.

(b) Now use the same technique on colorpeppersdata. This creates a black and white equivalent of the original color image of the peppers. View it with Image to verify the result, then Export the grayscale image to a “.jpg” file and open that file with external software.

Video

Mathematica can import and export a number of video formats, including AVI, FLV (Flash video), QuickTime, and animated GIF. The guide page on Multimedia Formats contains links to information on many of them. For this lab, we’ll focus on the single task of exporting animations to a video format which you might want to embed in a presentation or display on the web.

Exporting with Animate. Perhaps the simplest way to export data is to create an animation with Animate, then use the Export command. You can export to AVI, MOV, or SWF formats in this way (and probably others). AVI files are the largest but most universally accepted by programs such as PowerPoint.  MOV format files are much smaller and faster if your target software supports them.  SWF files are also much smaller and work well on web pages. 

Let’s create an animation of a traveling sine wave. (Try the Background-> White version if you have any issues with flickering.)

sinewave = Animate[Plot[Sin[2 Pi (x-t)],{x,0,2}], {t,0,2}]
(* sinewave = Animate[Plot[Sin[2 Pi (x-t)],{x,0,2},Background->White], {t,0,2}] *)

Here’s how you would store the animation as an AVI file. (This takes a bit of time, ~25 seconds on my fairly fast office computer.)

SetDirectory[NotebookDirectory[]]
Export["sine.avi",sinewave]

Verify that it worked by viewing the file using something like Windows Media Player or VLC Media Player. Some things might catch you by surprise, such as (a) the animation includes the (non-functioning) controls, and (b) the animation runs forward and then backward, instead of just forward.

Exporting by creating an array of images.  Although using the Animate command followed by Export is often the simplest way to export a video, it is not usually the best way. In fact, I myself never use it. A better way to create animations is to create an array of graphics objects and then export the array. That usually provides a smoother animation, a smaller file, and removes the two oddities mentioned in the last paragraph.

The next cell creates 41 different graphs (from t = 0 to t = 2, in steps of 0.05), then exports them into a single, merged avi file. Verify that the file sine2.avi gets created, and is viewable in e.g. Windows Media Player. (This takes far less time than the sine.avi animation done previously.)

sineWaveArray=Table[Plot[Sin[2 Pi(x-t)],{x,0,2}],{t,0,2,0.05}];
Export["sine2.avi",sineWaveArray, AnimationRate->10]

Assignment 4. Exporting an animation to a web browser

Create an animated gif (“filename.gif”) of the function f(x,y,t)=3 cos(0.2 t) exp(-(1/3) ((x^2)+(y^2))) for x and y between -5 and 5, and t going from 0 to 20π (in steps of 0.5). Use Plot3D to display the function. Use PlotRange->{All, All, {-3,3}}  as a plot option in order to preserve the z-axis scale from plot to plot. View your file in a web browser and show it to your TA.

Audio

Mathematica can represent functions not only as graphs via the Plot command, but also as sounds via the Play command. Listen to the output of Play in this example (use headphones if your computer doesn’t have speakers; you can borrow a pair from a TA or a fellow student if you don’t have your own).

concertA[t_] = Sin[2 Pi 440  t]  (* t assumed to be in seconds *)
Plot[concertA[t],{t,0,.05}]
Play[concertA[t],{t,0,2}]

Here’s a dial tone for you:

dialtone[t_] = Sin[2 Pi 440  t] + Sin[2 Pi 350 t]
Play[dialtone[t],{t,0,2}]

Sounds are elements of the Sound category. Play is only the most basic sound-related command. The guide page on Sound and Sonification contains links to most of the additional standard functions for creating and analyzing audio signals. If you have have a strong interest in music and/or audio, you may also want to explore the special Music and Audio packages. 

Here’s another example. This is just for fun, but why would a constantly increasing frequency sound like it does when Mathematica plays it?

expfreq[t_]=Sin[2 Pi 440 Exp[t]]
Play[expfreq[t],{t,0,6}]

As with numerical data and image data, one can Import audio data from disk or from the internet, and there are also many audio clips available via ExampleData.

Assignment 5. Deepening a French horn

(a) Find and load the FrenchHorn example data listed under “Sound”. Play the sound (the Play command is not even needed). Use InputForm to look at the metadata at the end of the list of numbers (delete after done viewing). There should be only a single item of metadata: the number 22050. That indicates the sampling rate (in Hz). It’s at the [[1,2]] position of the data, as you should be able to verify.
(b) Modify the horn sound by replacing the number 22050 with the number 11025. What happens to the horn sound, and why? (There are two effects.) 
(c) Export the new sound as a wav file, and verify that it plays in e.g. Windows Media Player.

Integrated Data Sources

A cool feature of Mathematica is its ability to interact with Wolfram’s collection of “curated data”. The curated data sets give you access to values from chemistry, astronomy, geography, the stock market, etc., and are actively updated by Wolfram’s staff as new/better values become available. See this website for a partial listing of available data sources: https://www.wolfram.com/technology/guide/LoadOnDemandCuratedData/  

The data sets are accessible in real-time over the internet without needing to specify any webpages or anything like that. Whenever you query one of the data sources, the requested data is automatically downloaded to your computer and imported into your notebook. While we only have time to explore two of the available data sources in this section, you are encouraged to explore some of the others later on your own.

Element Data

ElementData provides a variety of information on each element in the periodic table.  Evaluate the cell below, where we display a list of the elements in the database and a list of the properties that may be available for each element.

ElementData[]
properties = ElementData["Properties"]

To access a particular property, you just use the ElementData function, specifying the material and the property. For example, here are the atomic number and density of iron (in SI units).

ElementData["Iron","AtomicNumber"]
ElementData["Iron","Density"]
LinguisticAssistant  (* for this one I typed "<control-equals>density of iron" *)

Here’s a command to see all of the information on iron, all at once. It produces lengthy output so I recommend running the command in a new window.

MatrixForm/@{#,ElementData["Iron",#]}& /@ properties // TableForm; (* remove semicolon to display output *)

Not every property has been measured for every element in the periodic table.  You probably saw that some of the properties of iron were listed as Missing because they were either "NotAvailable", "NotApplicable", or "Unknown".  Here’s a function that will be useful to you--it eliminates any Missing items from a list of data.  We test it for one example to display all of the elements whose magnetic and electrical types are known.

skipmissing[data_] := Select[data,!MemberQ[Flatten[#],_Missing]&]

{#,ElementData[#,"MagneticType"],ElementData[#,"ElectricalType"]}&  /@ ElementData[];
% // skipmissing

Assignment 6. Melting and boiling points

(a) Create a 2D list that contains a pair of Kelvin-degree temperatures, the "AbsoluteMeltingPoint" and the "AbsoluteBoilingPoint", for each element in the database. (No need to have the element name in the list.) Then use skipmissing to eliminate the pairs that are Missing one or both of these temperatures, and employ N to ensure that all of the entries contain true floating-point data.
(b) ListPlot the resulting data with the following options:  PlotRange -> All, Frame -> True, FrameLabel -> {“Melting Point (K)”, “Boiling Point (K)”}.  What do you learn from this plot?

Astronomical Data

Evaluate the cell below to learn more about the information available through AstronomicalData.  The four commands display the total number of astronomical objects in the database, the object classifications used (e.g. planet, moon, star, galaxy, etc.), the properties stored for these objects, and finally an image of each “Planet” in our solar system (sorry, Pluto!).

Length[AstronomicalData[]]
AstronomicalData["Groups"]
properties =AstronomicalData["Properties"]
{#,AstronomicalData[#,"Image"]}& /@  AstronomicalData["Planet"]

Assignment 7. Hubble’s Law

(a) Create a 2D list called data1 that contains pairs of numbers: the “DistanceLightYears” and the “Redshift”, for every "Galaxy" in the AstronomicalData database (supress the output; it’s long).  Then use skipmissing to eliminate the pairs that are Missing one or both of these quantities, and employ N to ensure that all of the entries contain true floating-point data.  Name the result data1.  Once you have succeeded, suppress the lengthy output with a semicolon. Hint: Be sure to select for galaxies the same way I selected for planets in the example above. To check yourself, there were 729 useable galaxies in the curated data as of Jan 2014.

(b) The distances in data1 are measured in light years and the redshifts are unitless.  A more traditional description would express the distances in megaparsecs, and would multiply the unitless redshifts by the speed of light in kilometers per second.  We provide multiplicative conversion factors in the cell below.  Convert data1 to traditional units and name the result data2. Then ListPlot data2. You should find a nearly linear relationship between how far away a galaxy is (the x-axis, the distance in megaparsecs), and how fast it’s traveling (the y-axis, the redshift in km/s). That is Hubble’s Law, discovered by Edwin Hubble in 1929, and is strong evidence for the Big Bang theory (the scientific model, not the TV show). The slope of the line indicates the rate of the expansion of the universe. 

Hint: One way to apply the conversion factors is to make a nameless function that multiplies the first element of a pair by mpperly and the 2nd element of a pair by lightspeed, and then use /@ to evaluate this function for each element in data1. Alternatively, you could separate data1 into two lists with Transpose, multiply each list by the appropriate constant, then re-form your x-y pairs with a second Transpose command.

mpperly = 3.0660*^-7; (* Megaparsecs/LightYear *)
lightspeed = 2.9979*10^5; (* km/sec *)

Additional Assignments

Assignment 8. Calibrating spin resonance data

(a) The file spinres.csv, available on the Physics 230 website, contains data from an electron spin resonance experiment I did where the magnetic field was changed while we recorded an optical signal measuring electron spin polarization. The first column represents the magnetic field as recorded by a magnetic field sensor; the second column is the optical signal. Read in the file and ListPlot the data. Be sure to use the PlotRange->All option. Hint: since it’s in a nice csv format, you can read the data in with a simple Import comand without any options.
(b) The y-axis scale isn’t too important, but the x-axis scale is critical. The problem here is that the x-axis is not really displaying magnetic field (which should range from 0.718 to 0.841 T), it’s displaying the voltage from my field sensor (which ranged from -1.207 to -1.157 V). We need to therefore convert the magnetic field sensor data into tesla and replot. In a separate experiment, I calibrated the magnetic field sensor so I would know what voltage from the sensor corresponded to what field. The calibration information is in in a file called calibration.csv; the first column is voltage and the second column is the field (in tesla) that a given voltage corresponded to. Read in the calibration information, use it to define a calibrating function (via Interpolation), and use that to change your x-values into tesla. Then make a ListPlot of the optical signal vs. actual field in tesla. This plot should look like a reversed version of the first plot, but with the proper magnetic field range (0.718 - 0.841 T) along the x-axis.

Wrap-up

The following functions were discussed in today's lab, or are very closely related:

See all Mathematica functions: Names 
Load package: Get (shortcut <<), or  Needs 
Packages: Units  (with Convert, SI, and CGS commands),  Physical Constants 
Directory/file-related commands: $InitialDirectory, Directory, SetDirectory, FileNames, CopyFile, DeleteFile, FilePrint 
Import/export of expressions:  << (Get) and  >> (Put)
Import/export of data:  Import, Export, ReadList, Read
Creating a function from a set of {x,y} pairs: Interpolation 
Viewing data in a table: TableForm
Viewing strings with quotes around them, or image/sound data in numerical format (with metadata): InputForm
String conversions: ToString (number or expression to string), ToExpression (string to number or expression), Characters (string to list of individual characters)
String functions: String, StringLength, StringReverse, StringInsert, StringReplace, StringPosition,  <> (StringJoin), StringSplit 
Merging numbers into strings: StringForm (often with NumberForm)
Input streams: OpenRead, Skip, ReadList, Close
Data for testing:  ExampleData
Image functions: ImageData 
Audio functions: Play, Sound 
Misc array function: PadLeft, ReplacePart 
Two of the integrated data sources: ElementData, AstronomicalData


Potentially useful summary/tutorial pages: 

Guide: File Operations
Tutorial: Units Overview
Tutorial: Reading and Writing Wolfram System Files 
Tutorial: Naming and Finding Files
Guide: Importing and Exporting data
Guide: String Operations 
Guide: String Manipulations
Tutorial: Reading Textual Data
Guide: Listing Of All Formats
Guide: Multimedia Formats 
Guide:Sound and Sonification 
Guides: Music and Audio packages
Wolfram website on curated data: https://www.wolfram.com/technology/guide/LoadOnDemandCuratedData/