[Book Summary - CtDSI] Cracking the Data Science Interview Ch. 1

 

1. What is Data Science?
   • definition: deriving insights from data. these insights are then used to provide value to a business
   • main factors
     • explosion of data (like MapReduce)
     • technological advances (like GPUs)
     • success stories
2. Data Science ≠ Machine Learning
   • Machine learning is based on the idea that algorithms can identify and learn patterns from data and make decisions with minimal human intervention
   • two diciplines that provide suitable tools to help solve data science problems
     • Operations Research: problem-solving techniques ranging from mathematical modeling and stochastic processes to mathematical optimization to simulation in order to improve decision-making
     • Statistics: for collecting, analyzing, interpreting and presenting empirical data
   • The most important for a data scientist to have a solid foundation in exploratory data analysis, data visualization, probability and statistics, optimization, mathematical modeling and computer science!
3. What Makes a Good Data Scientist?
   • identify the relevant questions
   • acquire and clean the right data
   • analyze that data to obtain results
   • clearly communicate the findings
   • conver the results into solutions
4. The Data Science Process Workflow
   • Specify Objective ⇒ Data Acquisition ⇒ Explore the Data ⇒ Establish a Baseline ⇒ Model the Data ⇒ Analyze Results ⇒ Communicate Findings ⇒ Iterate
5. Data Science Deliverables: Prediction, Forecasts, Anomaly Detection, Recognition, Optimization, Segmentation, Recommendations, etc.
6. Writing a Great Data Science Resume
   • One Page
   • Relevant Coursework
   • Relevant Skills
   • Relevants Projects: try not to include the common projects everyone has worked on
   • Relevant Experience: relevant experience and employment history with impactful bullet points
   • Include Accomplishments (put NUMBERS into your resume!): specific impact, business impact, competition ranking etc.
   • Customize: customize your resume to specific jobs
7. Data Science Interview Topics
   • Probability & Statistics: Conditional probability (Bayes’ Theorem), Probability Distributions, Hypothesis Testing, Covariance and correlation
   • Computer Science: Coding (Python or R), Data Structures (Lists, Hash Tables, Stacks, Queues, Treesm Graphs), Algorithms (searching, sorting, graph traversals), Databases (SQL, NoSQL), Distributed Computing (MapReduce, Spark, Hadoop)
   • Machine Learning: Supervised Learning, Unsupervised Learning, Deep Learning, General Predictive Modeling (choosing the right evaluation metrics, train and test sets, cross-validation)
   • Data Engieering: Data Wrangling, Cleaning and Visualization, Feature Engineering
   • Doman Knowledge: depneding on the company and industry
   • Behavioral and “Fit” Questions
8. Data Science Interview Process
   • Coding Challenge ⇒ HR Screen (asking you behavioral questions), Technical Screen (questions ranging from computer science to machine learning to statistics) ⇒ Take Home Project (testing your coding, analytical, and communication skills - Be aware of the target audience!) ⇒ Onsite ⇒ Offer & Negotiation
9. Behavioral & Fit Questions: Teamwork, Ability to Adapt, Communication using the STAR (Situation, Task, Action, Result) method

[Book Summary - Information Theory] Information Theory: A Tutorial Introduction, Ch 5 by James V. Stone

1. Entropy: Average surprisal 

2. Discrete variables  

• Continuous variables: divide the continuum into bins  
• Depending the number of bins, entropy changes
• If the number of bins increases, thus each bin size being smaller, entropy increases due to more options being made.

• Differential entropy
• Transforming continuous variables
• Changing the range of a discrete set of variables doesn't change the entropy
• Example: In the case of a binary dice, which features only 0 and 1, the output should be 0 or 1 even though either of them is divided by infinite numbers between 0 and 1 
• Changing the range of a continuous variable does, because the entropy is based on bin-width
• Doubling the range -> doubles the number of bins -> adds one bit, even if half of the bins aren't used
• What about adding a constant? 
• As the term "constant" implies, It doesn't change entropy because range (variance) remains the same. 
• Maximum entropy distributions
• What distribution can we engineer so that entropy is highest? 
• Fixed upper/lower bounds 
• Fixed mean, with all values >= 0 ==> exponential 
• Fixed variance (e.g., power) 
• Back to differential entropy
• Infinitely accurately…
• What in practice limits 'bin sizes'?
• Noise!
• If the amount of noise increase, it is getting harder to know what the actual signal was
• How does noise limit bin-sized connection to transmission of information in language?
• More noise, less precision due to the number of bins being smaller because of noise (i.e., log1/delta x decreases)
• Zipf's law: Infrequent words => longer, lexicon is limited        

• Bin size => Amount of signal! 

[Book Summary - Information Theory] Information Theory: A Tutorial Intorudction, Ch 4 by James V. Stone

     1. Mutual information 

• The average surprisal of the symbols in the system 
• The entropy of a single symbol = its surprisal 
• Conditional entropy H(X|Y) = noise entropy H(Y|X) 
• Accuracy of transmission --> mutual information (overlapping part is information transmitted)

• 2. Joint probability of two systems 

• 
  

• 3. Mutual information and information rate

• For a channel with inputs from (X) and outputs from (Y), the information rate is determined by their mutual information, which is in turn related to H(X), H(Y), and their relationship  
• What's the situation with: 
• High vs. Low input H? input entropy 6; output entropy 3 
• Let's think about tossing a dice and assume that we consider either odd numbers only or even numbers only; it should be wasting information without noise cases
• High vs. Low output H? 
• High vs. Low independence of input and output?
• If totally independent, there should be no overlapping portion in a Venn diagram.
• The input/output entropy should be high, and the noise should be low   

• 4. Crosstalk 

• A case of loss information: Substitution error
• Adding "redundancy" (help compensate for information loss) might be one way to solve such loss

• 5. Redundancy is good and bad?

• Increase the rate at which information can be transmitted (Resource cost; e.g., articulatory effort)
• But increase the accuracy with which it is transmitted  

• 6. Noisy typewriter example

•  

• 7. How to add redundancy?

• Adding more values
• You can adjust text as well
• Linguistic examples
• VC[+nasal]: 
• Vowels become nasalized
• This nasalization is redundant because vowels are already nasalized
• Thus, nasality could add some noise to the vowel (F1 distinction)
• English case:  
• In the case of word 'thank,' it seems that vowel is a bit raised because nasal [n] raises the vowel height, which eventually affects perceptual space
• Just Noticeable Difference (JND):  But think about whether there is anything that is totally free of redundancy. Where is the boundary? 
• English vs. Spanish Vowels: In terms of the number of vowels, Spanish is simpler than English, meaning that assigned space per vowel is broader than that of English. English can distinguish more vowels than Spanish, but it is costly.

[Book Summary - Information Theory] Information Theory: A Tutorial Intorudction, Chs 1-2 by James V. Stone

1. Useful information 

1. • (maximize) informative redundancy 

   • (minimize) non-informative redundancy 

2. Noise 

3. Minimizing "effort" required for transmitting each portion of an informative signal: selection in biology/language/culture 

• What is "effort"?  

1. i) articulatory? how much effort? (e.g., Spanish vs. English vowels) 

   i) cognitive effort? 

   Iii) motor plan for producing similar sounds, 

   v) sounds including some noise 

4. Bits and binary digits 

• Bits: splitting the possible (meaning) space in half / quantity of the info 
• Binary digits: each place in a binary number (a number consisting of binary digits) is 1 bit. 
• Bytes: 8? 

m outcomes = 2^n binary choices 

n binary choices = log2(m outcomes) 

n bits = log2(m outcomes) 

 

 

5. Random variable: e.g., X(head) = 1, X(tall) = 0 / a kind of function  

6. Rather than an actual variable ? 

7. Schema for message transmission: a message is an ordered sequence of symbols corresponding to outcome values of random variables 

• Data: meaning… social… discourse… / linear symbols…? /  
• Encoder: codewords, taken from codebook (cf. symbols vs. codewords: might be same, but not necessarily, for example, freq.words with min.symbols & infreq.words with max.symbols 
• Input x: can be identical to message, or transformed by some mapping (e.g., to compress removing redundancy) 
• Channel: noise…? (can obscure the speech - brain filters the noise…?) need to know the differences btw noise and redundancy 
• With regard to top-down processing: redundancy is useful in that it confirms… (e.g., los angeles) 
• Output y: lossless, lossy 
• Channel capacity: information per codeword; information rate = information per sec / close to the channel capacity…? 
• Fewer contrast: by position to position fast but needs more time  
• More contrast: by position to position slowly but needs less time 

• Noise reduces channel capacity

8. Surprisal and entropy 

• Surprisal = log(1/p(x)), or - log(p(x)) 
• If we use log2, surprisal is measured in bits. 

9. Entropy 

• Average surprisal of elements in a system: H(X) 
• How do we estimate this in practice? 

10. Entropy of a system

11. Some assumptions

• Independent and Identically Distributed (a.k.a. i.i.d.)
• Stationary: doesn't change through time
• Ergodic: system can be represented by a reasonably large number of measurements 

12. Uniform Information Density Hypothesis: if language is optimized for information transmission, it should tend toward equal surprisal for symbols per unit time 

Discussion: The actual entropy of written English letters lower than that of English estimated from the individual prob of letters from a big sample of text? Cuz that is not independent! More skewed because of the context!