![[personal profile]](http://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
just found a bug...
Sep. 21st, 2017 03:01 pmIn spark.core, RNG, specifically, normal distribution RNG.
It caches values (randomly). Now try to reseed.
It caches values (randomly). Now try to reseed.
just learned that...
Jun. 14th, 2017 11:12 am"Spark is faster than map/reduce". A guy working at Databricks is giving a talk regarding how Spark works.
Omfg; how can one possibly deal with all this?
UPD. From the same talk.
Omfg; how can one possibly deal with all this?
UPD. From the same talk.
cartesion()in case you are interested in Spark...
Jan. 31st, 2017 03:04 pm
// Set name from main class if not given
name = Option(name).orElse(Option(mainClass)).orNull
if (name == null && primaryResource != null) {
name = Utils.stripDirectory(primaryResource)
}
Just a chunk of their funny shitty code.
FYI,
var name: String = nullI'd rewrite all that crap, but it's not only code, I'm afraid. It's the whole Spark world that needs a doctor.
spark masterpieces
Sep. 26th, 2016 03:42 pm/** * Returns a Seq of the children of this node. * Children should not change. Immutability required for containsChild optimization */ def children: Seq[BaseType] lazy val containsChild: Set[TreeNode[_]] = children.toSet
Imagine, we override the class, but our
children are not to be mutable. Why t.f. then not declare it val?!Another masterpiece
object CurrentOrigin {
private val value = new ThreadLocal[Origin]() {
override def initialValue: Origin = Origin()
}
def get: Origin = value.get()
def set(o: Origin): Unit = value.set(o)
def reset(): Unit = value.set(Origin())
def setPosition(line: Int, start: Int): Unit = {
value.set(
value.get.copy(line = Some(line), startPosition = Some(start)))
}
def withOrigin[A](o: Origin)(f: => A): A = {
set(o)
val ret = try f finally { reset() }
reset()
ret
}
}
So, we have a static object that wants to do something with a context. (See
withOrigin.) That's "dependency injection", Fortran style. And we can do without vars, because, well, it's ThreadLocal.Who wrote all this... could they hire Scala programmers, I wonder...
an amazingly funny piece of code
Jul. 30th, 2016 07:42 pmhttps://github.com/apache/spark/blob/
So, spark gets a function to apply to data (e.g. in foreach, or in filter). Before running it, it (like a raccoon) cleans the function.
Enjoy the nice reading.
ClosureCleanerSo, spark gets a function to apply to data (e.g. in foreach, or in filter). Before running it, it (like a raccoon) cleans the function.
Enjoy the nice reading.
